Mosaic Unwrap -> CUDA Error Out Of Memory

[WillBarley]

Hey guys, I need some help with Mosaic Texturing.

I cannot perform Mosaic Unwrap
I consistently hit a CUDA Error that says Out Of Memory

I even tried setting the quality to 1%, but still hit this error.

Steps to get here:

• clear cache by manually deleting all of it
• Reality Capture 1.4
• reset application to “Clean Install”
• import 1500 100MP JPGs
• create a Normal mesh model
• attempt mosaic unwrap with the following settings
  
  

fails with this error

Try again to unwrap:

• clear cache by manually deleting all of it
• Reality Capture 1.4
• reset application to “Clean Install”
• attempt Mosaic unwrap with the same settings but quality reduced to 1%

same error:

It looks as if the system runs out of memory each time

What can I do to get this to work?

I;m using 8 x Nvidia L4 GPUs with CUDA 12.4

[OndrejTrhan (otrhan)]

Hi @WillBarley
My colleague tested this using the similar dataset and he was able to create a texture.
Can you try to use other style in the unwrap settings?
Also, using the other unwrap method is working for you?
Can you check the CUDA driver on your machine?
Can you also try to disable some of the GPUs and keep for example just two of them for RealityCapture?

[WillBarley]

Thank you for your reply @OndrejTrhan

I will try what you suggested

• enable only 2 GPUs in RC
• use a different Quality for Mosaic Unwrap

Here we go:

And you can see that only 2 GPUs are enabled now:

Running now…

Failure during unwrap.

I’ve not seen the “102” number before -what does it mean?
Why does it suggest updating my CUDA driver? I’m on a pretty recent version.
Any idea what’s going wrong @OndrejTrhan ?

Yes,
Geometry unwrap works

Check CUDA drivers - what would you like me to check?
In my post you can see the CUDA driver version (12.4) - what else can I check?

[OndrejTrhan (otrhan)]

Hi Will,
I found this about the mentioned errors:
CUDA_ERROR_OUT_OF_MEMORY = 2
The API call failed because it was unable to allocate enough memory or other resources to perform the requested operation.

CUDA_ERROR_DEVICE_NOT_LICENSED = 102
This error indicates that the Grid license is not applied.
Regarding this error there are some posts, like: Unable to retrieve a GRID vGPU licence - XenApp - NVIDIA Developer Forums

Ca you also try to use only one GPU (as a test case)?
12.4 is a CUDA toolkit version, there should be already a newer one. And what I meant is the driver version (Download The Latest Official NVIDIA Drivers). It is possible that using the newest or one of the older will help.

Is this error also happening on other dataset? Can you try to simplify your model and unwrap that smaller model?

[WillBarley]

I will try those things.

Is there a way that I can lookup those error messages like you?

On this one:
CUDA_ERROR_OUT_OF_MEMORY = 2
The API call failed because it was unable to allocate enough memory or other resources to perform the requested operation.

Is this GPU memory or system memory? I usually see system memory rise to 100% and then the process fails. So I have believed that it is system memory that is running out. Is that correct?

[WillBarley]

I’ve done some test and it seems that switching to
Unwrap Style = Maximum Texture Count
and setting the max count to a very high number
Seems to allow us to successfully use Mosaic to do Unwraps and Texture.
I’m running some more test now to confirm.
Good News!

@OndrejTrhan I also noticed a bug:
using the -unwrap command ignores the current unwrap settings. You can see it if you do this:

-set "unwrapMinTexResolution=16384"
-set "unwrapMaxTexResolution=16384"
-unwrap

Then notice after unwrapping that the size of the unwrapped texture is not 16384.

[WillBarley]

I hit another CUDA error: 2 - what is it indicating?

• I don’t see high GPU memory usage at any point,
• I do see 100% system memory use at one point

on the GPU metrics,

• green = the average level across the 8 GPUs
• purple = the maximum level across the 8 GPUs

…gathering more information…

…thoughts…

• maybe it is the GPU that is OOM’ing.
• I can see that at least 1 GPU ends up with a lot of memory allocated and it stays there until you restart Reality Capture or start a new process in Reality Capture
• Maybe I don’t see 100% GPU memory usage in the chart because it happens quickly: the chart gets a data point every 10 seconds, perhaps it OOMs in a 10s period between data points.
• why would it OOM? Perhaps something tries to allocate “all free GPU memory” on one of the GPUs, while at the same time some other process is using the GPU… i’m not sure how that kind of race condition would occur, but maybe so. What other processes are using the GPU? other apps displaying things on screen because the GPU is in WDDM driver model so that it can render to the screen in Reality Capture - so perhaps putting them in TCC mode and perhaps running headless would help avoid a race condition?

[OndrejTrhan (otrhan)]

Those errors number are from NVIDIA errors page, so I suppose they are connected to the GPU.

That is not a bug. There is texture’s unwrap and unwrap’s unwrap. These settings are got texturing. If you want to change unwrap’s you need to use XML file (Texture unwrap settings are not applied - #9 by OndrejTrhan).

[WillBarley]

Thank you @OndrejTrhan