Hi,
after out upgrade from UE5.5 to UE 5.8 we’re encountering a pretty random crash when creating RTSPOs (due to lumen activated via PPV) in our testing pipeline. According to the callstack this is because a call to a function in NVAPI which returned -2 instead of NVAPI_OK, which causes an ensure or an assert which crashes the engine.
In the logs I noticed every time the crash occurred, there was this line, sadly without a reason why:
LogD3D12RHI: Warning: Failed to initialize NVAPI
This happens on g5 instances (Nvidia A10G) on WIndows Server 2022 with the current Nvidia drivers installed (596.36).
I wasn’t able to reproduce it yet, so I don’t really have much more info, a log and the crash dump are attached.
Is this a known issue with the driver on cloud hardware? Please let me know what you know or what additional information you need. I’ll update the ticket if I encounter more details.
Cheers,
Simon
[Attachment Removed]
Steps to Reproduce
Sadly I wasn’t able to reproduce it yet.
[Attachment Removed]
Hi Simon,
I think the Lumen RTPSO compilation error is a red herring and unlikely to be the cause of your crash. The NVAPI not being able to be initialized is likely the main problem. Could you make a code change at WindowsD3D12Device.cpp:1360 and print out the exact error code that NvAPI_Initialize() returns? That way, we can get more info on why your A10G cannot initialize properly.
Cheers,
Tim
[Attachment Removed]
Yeah, unfortunately, this is not a known issue for us. The assert indicates that something happened that prevented the nv_api.dll from loading, but that’s about it. Do you see anything in the server’s Event Viewer that relates to this assert?
[Attachment Removed]
Hi Simon,
I am glad you were able to sort that issue out! I’ll mark your answer as best, so that others who might be having the same issue can take your resolution as advice. Feel free to let us know in case you run into anything else.
Best,
Tim
[Attachment Removed]
Hi Tim,
thanks for the reply.
Our pipeline depends on a vanilla Unreal Engine from the Launcher, so manipulating the Engine source is only feasible locally. Unfortunately I have a very hard time to reproduce it locally on such an instance.
Is there something else I can check without having to change the engine source?
Maybe some more context in case this might be relevant:
- We’re running tests using Gauntlet, running with -renderoffscreen. Everything is orchestrated by by scripts of our pipeline. When I try to reproduce it on an instance I connect via remote desktop, maybe this somehow prevents the error.
- There are cases where only an ensure is triggered without a crash:
AVE_Application +0x10becd51 UE::Assert::Private::EnsureFailed
AVE_Application +0x10cb4a13 `FD3D12RayTracingShaderBindingTableInternal::CompareData'::`5'::<T>::operator()
AVE_Application +0x093da8d4 FD3D12PoolAllocator::CreatePlacedResource
AVE_Application +0x093e0af9 FD3D12RayTracingPipelineCache::FShaderCompileTask::DoTask
AVE_Application +0x093e30f9 TGraphTask<T>::ExecuteTask
AVE_Application +0x01b27ae9 UE::Tasks::Private::FTaskBase::TryExecuteTask
AVE_Application +0x01af909f LowLevelTasks::TTaskDelegate<T>::TTaskDelegateImpl<T>::CallAndMove
AVE_Application +0x01b04edb LowLevelTasks::FTask::ExecuteTask
AVE_Application +0x01b04c8c LowLevelTasks::FScheduler::ExecuteTask
AVE_Application +0x01b2b91d LowLevelTasks::FScheduler::WorkerLoop
AVE_Application +0x01af15f9 operator<<
AVE_Application +0x01d9d543 FThreadImpl::Run
AVE_Application +0x02211987 FRunnableThreadWin::Run
AVE_Application +0x0220b384 FRunnableThreadWin::GuardedRun
KERNEL32.DLL +0x00014caf BaseThreadInitThunk
ntdll +0x0007edea RtlUserThreadStart
I was hoping this is an already known issue 
I will continue to investigate and try to reproduce with your suggestion, any additional hints or strategies to get to the bottom of this are appreciated.
Cheers,
Simon
[Attachment Removed]
I will take a look and keep you posted.
[Attachment Removed]
Hi Tim,
we’ve finally identified the issue and it is not engine-related. The crash with 5.8 is a symptom, which didn’t occur with 5.5. So we had the issue for quite some time but didn’t notice. Not only the nvapi.dll was missing, but also nvcuda.dll, nvcuvid.dll and nvEncodeAPI64.dll. This led me to believe it was a driver install issue in our test pipeline.
In case others stumble over a similar issue here is what caused the problem:
Out pipeline starts cloud instances to run functional tests. To ensure the GPU driver is always up-to-date it was installed right after startup with the -clean parameter. G5 instances with their A10G GPU seem to take a little bit longer than other instance types until the GPU is ready, so installing the driver too soon failed. Due to the -clean parameter the affected DLLs were deleted, but not re-installed due to the error we didn’t catch. A simple delay between instance startup and driver install seems to fix the issue.
The issue can be closed, thanks for every hint and nudge in the right direction!
Cheers,
Simon
[Attachment Removed]