If you’re using ray tracing then you don’t need to use lightmaps, ray tracing is completely dynamic so it’s rendering the image every frame.
If you’re using GPU lightmass (where it’s baking lightmaps and it’s accelerated by your GPU) Then it’s limited by your GPU memory (8GB) and it has to load all of the geometry and textures (including the lightmaps it’s going to render). So it’s very easy to use up the GPU memory that way.
If you don’t use GPU lightmass then it can render with the CPU and system memory, since you have 64GB it should work fine that way.