Hey there, and thank you for posting on our community 
Let me try to see if I can help you 
“The CUDA error 700 (‘illegal memory access’) usually indicates that your GPU tried to access memory it wasn’t supposed to. This can happen if the system runs out of memory or if something crashes during GPU operations, and unfortunately, it can sometimes corrupt the CUDA context. That’s why even going back to ‘Normal Quality’ still throws the same error, the GPU is still in an unstable state.”
Here are the steps I would recommend trying for troubleshooting if you haven’t already:
- Restart Application ( if that fails, see number 2)
- Restart your computer to make sure that it clears the CUDA context and resets the GPU.
- Make sure you have the latest version of RealityCapture
- Check GPU memory usage. In this case GTX 1660 has 6GB VRAM, which is only enough for a small/ medium dataset
Cheers
MC