Extremely basic project gets '‘Video memory has been exhausted' message

This means that the foliage is not optimized for the best frame rate. Commonly happens with realistic looking foliage, a scenario when you want to use Nanite, but then you probably need foliage which is created for Nanite (see below video). Your forest scene performance can likely be improved by adding a culling volume to the scene, and or culling settings of each foliage mesh, individual mesh reduction settings, lower texture resolution, imho max 2K, and the foliage has billboards (unless it is Nanite).

This week I upgraded from 5.5 to 5.7 and to my surprise also encountered video memory has been exhausted, which suggests that in 5.6 or 5.7 something changed. But even without this crash, I had a user reporting that he crashed with the build but could not figure out what it was.

Even if you manage to adjust your project it could cause issues for others with different hardware. But the route to optimization is not a straight forward one. Dallas summarized it like this

Imho, basically if you strife for a realism looking product your way is to use the new UE5 features like Nanite. And at first most Unreal developers probably thought this is the way to go, the next logical step. But it is not needed for I guess the majority of projects, and when you as highlighted want the best FPS.

And if you want the best graphical fidelity when using Nanite you need probably want real Nanite foliage.

And as I recently learned from Ardyn at the SoStylized Discord, this tool is pretty handy in terms of merging meshes, imho should become an Engine feature.

And what has been less communicated if you use Nanite and you encounter crashes it may be prevented by reducing the Nanite Triangles, each mesh has this Reduction option.

Notice: If you are using Nanite and plan on merging meshes, check the current recommendations. And HLOD is a mesh clustering feature which becomes available when your level is setup using World Partition.

Nanite assemblies work by taking the clusters of the part meshes and encodes them into the final mesh’s hierarchy without duplicating the geometry of the parts during build time. At runtime, Nanite handles the part instances by calculating their final transform on-demand during cluster culling as it descends the hierarchy.

On non-Nanite platforms, the triangles of the parts are transformed and merged together into a single mesh. From there, it is simplified to generate a fallback mesh that works the same as regular Nanite fallback meshes. Nanite Foliage | Unreal Engine 5.7 Documentation | Epic Developer Community

So there are lots of techniques and features to optimize a project but first you have to decide what you are aiming for.