CUDA error : 2 : out of Memory

Development Asset Creation
,
1

specs

Processor:

Memory: 16.0GB

Graphics Cards

OS:

The Error  : CUDA error : 2 : out of Memory

 

This happens every time I go to reconstruct anything other than the preview reconstruction.

Things I have tried

 

Things to note about the setup

Reconstruction Settings

  • have set both GPUs to use and every combination inbetween
  • Normal Model Image downscale: 5
  • Mesh calculations: anywhere between 0 - 10
  • I have left macimal vertex count per part at: 5,000,000
  • Detail decimation factor: anywhere between 0 - 100

I have not changed or messed with anything in the Advanced settings.

 

I hope this give you enough information about the issue and what I’ve done to try and make a reconstruction happen. If there is anyone with more information on what I should try next or has a setting I should change I would love to hear from you!

 

2

got same problem today   454 pictures  only  imported form video  and when trying to generate  high model i 'm getting this error message     

and i’m using    gtx 970 + gtx 660ti in my pc    with i7-3820  and 64 gb ram 

3

Hello dear users

We we recommend to only use 1 GPU while working with RealityCapture currently. Try to do so and report back if this helped you.

 

4

I have tried with one card and that did not change the result. Also want to be clear, do you mean take the graphics card out of the machine or make sure the settings are set to only use one? I have changed the settings and got that same result.

5

I’ve run a few more tests and i’m now getting a crash when running normal reconstruction. it jumps between this and the CUDA error

6

Got the same problem today after updating to a version fixing an issue with 2D view of laserscan pointclouds. No difference using just one or both GPUs. 

2x GTX 1080

3000 images and about 100 laserscans

7

Exactly there is no difference if you will use one or two GPU units and support only advice o use one graphic card. Besides all my scans was done with high detail on two GPU units until installation of new version of reality capture :frowning: so definitely it’s a software issue in new version

8

@ any word on how we should be proceeding?

9

Hello,

This is our official statement for this issue.

 

Two immediate temporary solutions:

  1. Physically remove all GPU cards, except one - single card configuration.

or

  1. Install old GPU driver. The version 419.67 is working for us.

Reason:

The issue is with the CUDA memory de-allocation function, that has stopped working properly with latest NVIDIA GPU drivers.

More specifically the function CUDAFreeHost() resulted with success code, but the memory was not de-allocated and therefore after some time, the GPU pinned memory was filled up and the SW ended up with the message “CUDA error : 2 : Out of memory”.

It seems that this bug causes problems also to another CUDA developers:

https://devtalk.nvidia.com/…/cudafreehost-not-clearing-all…/

We have contacted NVIDIA and we hope that there will be a new driver with fix very soon.

Meanwhile, we are also trying to fix the problem by some workaround in our code and we will try to release hotfix ASAP.

10

thank you, Going to the 419. 67 version of Nvidia drivers worked for me I guess I wasn’t going back far enough in the archive.

11

That’s great news Mr. Keutzkamp. Good luck in your projects and enjoy RC.

12

Is 419.67 still the recommended driver version for the latest release of RC?

13

With the latest update, you should be able to run with the newest drivers. If you anyways experience any of such issues try rolling back to the 419.67s, I do believe you will not have to though.

14

Hi, I have this CUDA error popping up once in a while without any clear logic - is sometimes appears, sometimes not -no matter if the project has 40 or 900 images - but when it does, it is usually during the texturing process and it keeps reappearing repeatedly even after canceling the error window many times - so as the result the texturing process could not be finished…

I have tried the newest Nvidia drivers, and the older one recommended above (419.67) but it still keeps appearing, can you help me solve this, please???

PS I do not have SLI - just one older GeForce GTX 750 card…

15

Dear user,
this error is not RC related and aims directly towards Nvidia as your GPU, it probably has some issues with memory leaking or use. Sorry to see that, but I don’t think I can help you here. 

16

Sorry for posting in this old thread, I just came across the same memory problem and didn’t want to open a new one just to post what helped me fix it:

I’ve got a GTX 1050Ti and a GT 710 that I only use for extra monitors. I’ve reconstructed parts of the same mesh multiple times before but when I started the full reconstruction in high detail I got the same “low device … memory” error message. It looks like RC ran out of memory because it was using the 710 instead of the 1050 (which should be the main one).

You can choose the GPU RC uses in:

Reconstruction - Process / Settings - Image depth map calculation / GPUs to use

I set the 710 to “False” and it looks like it’s working fine now. :slight_smile: