FYI to others. I was having this issue (3080Ti Win11, UE5.1). Tried a number of things, including DDU’ing my drivers, changing recommended windows timeouts, etc. In the end it turned out somehow a setting in my BIOS got changed that was messing with my RAM memory voltage/timings.
I discovered this by running the built in “windows memory diagnostic” which failed and claimed I had hardware errors. This lead me to discovering and setting the BIOS settings back to reasonable values. I re-ran the memory checker and it successfully completed multiple passes. Haven’t had the issue since.