Open the Windows command prompt, then run where.exe dotnet – it will return one or more paths containing a dotnet.exe executable.
If it looks like, for instance:
C:\Program Files\dotnet\dotnet.exe
c:\program files (x86)\dotnet\dotnet.exe
…then you are okay. If the (x86) line is first, however, that will cause UE5.1 to use the 32-bit version of the dotnet executable… and it will run out of memory during the build process.
If that’s the case, you want to go to your computer’s settings and switch the order of paths so that the 64-bit one (e.g. the one without (x86) in the path) comes first.
Open Explorer, go to “This PC” (or “My Computer” or whatever), right-click and get “Properties”, and then click on “Advanced System Settings”. From there, click on “Environment Variables”, scroll down until you find the variable named Path and edit it appropriately.
If things are already in the right order, you may have some other underlying issue (like genuinely not having enough memory).