Hi Will,
I found this about the mentioned errors:
CUDA_ERROR_OUT_OF_MEMORY = 2
The API call failed because it was unable to allocate enough memory or other resources to perform the requested operation.
CUDA_ERROR_DEVICE_NOT_LICENSED = 102
This error indicates that the Grid license is not applied.
Regarding this error there are some posts, like: Unable to retrieve a GRID vGPU licence - XenApp - NVIDIA Developer Forums
Ca you also try to use only one GPU (as a test case)?
12.4 is a CUDA toolkit version, there should be already a newer one. And what I meant is the driver version (Download The Latest Official NVIDIA Drivers). It is possible that using the newest or one of the older will help.
Is this error also happening on other dataset? Can you try to simplify your model and unwrap that smaller model?