Video memory has been exhausted

You are probably out of VRAM because you only have 3GB of VRAM lol - the gtx 1060 does not meet the minimum specifications for UE5. The new features hog a LOT of VRAM.

Switching to DX11 probably solves it because the new features require DX12, and you are indirectly disabling the engine from utilizing Lumen and Nanite.

If you don’t plan on upgrading hardware you may want to consider sticking with UE4, since your PC will manage a lot better with Baked lighting (I know the 4.26 UI is a lot worse than UE5, feels very outdated, but it is a much more stable platform and better performing especially on older hardware).