Error: run out of video memory while building HLODS

This thread may help

Otherwise, try this, may not help with your situation, but worth a try with any HLOD problems, make a new HLOD file in content browser, delete your old one, replace references with your new one on the first window, but don’t save anything when the next one pops up.
Then build Landscape (ignore the 1000’s of yellow warnings, filter them to see errors) and the build HLODs
Delete HLODs via the Build menu doesn’t always work, this gives it a fresh start.

What is your cell size and loading range ?
Are most of your props added to HLOD or just the props that can be seen from a distance ?