Course: Neural Network Engine (NNE)

gabaly92 · April 12, 2024, 7:20am

Hi all, I have this error that is driving me crazy, I can’t figure out what it is, it does not make any sense.

I am running a model that generates an embedding from an input text in NNE, here are the model inputs and outputs descriptions.

Inputs - 3 tensors of type int64
Outputs - 2 tensors of type float32

I tokenize the sentence and then create the 3 input bindings and set the input shapes
then I set the output shapes as well, all based on the number tokens

here is the weird part, inference runs fine and the outputs are correct, but unreal engine just decides to crash randomly after all the code runs properly at the end of BeginPlay, like one time it runs fun and then the second time I run the level, it crashes with a memory access violation 0xffffffffffffffff why is there a memory access violation after all the code runs fine ? is it the garbage collector ?

The class that handles the network is an actor component implemented in C++, and added to a simple actor blueprint, if anyone has any ideas on why this might be happening or had the same experience with the engine before please let me know here is the error

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0xffffffffffffffff

UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_OutputLog
UnrealEditor_OutputLog
UnrealEditor_OutputLog
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_SlateCore
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

Thank you

gabaly92 · April 19, 2024, 8:59pm

@ranierin Does NNE have a problem with multi output Neural Networks ?, I just tested a Vision Model with 4 outputs, again the outputs the correct but Unreal Engine decides to crash again with a memory access violation

Thank you

ranierin · April 24, 2024, 5:59am

Hey @gabaly92 ,

NNE should be able to handle multiple input and output tensors, we have some multi output models in our testbench and some internal use cases are using multiple outputs too. However, there is always the chance that there is a bug.

Are you working with async tasks / multi threading? Are making sure that input and output memory is owned through the lifetime of the inference?

Another possible issue could be int64 not being handled correctly, is it possible that you try to adapt the model to consume int32?

Which runtime are you using?

mattai · April 26, 2024, 1:22am

just a quick note to say that the NNE API has changed in 5.4 so if you were using NNE with 5.3, you will need to update your code (and the documentation hasn’t been updated yet, so take a look at the source).

Here are some pointers to help quickly update:

TArrayView<TWeakInterfacePtr<INNERuntime>> Runtimes = UE::NNE::GetAllRuntimes();
for (int32 i = 0; i < Runtimes.Num(); i++)
{
	if (Runtimes[i].IsValid()) {
		if (auto CPURuntime = Cast<INNERuntimeCPU>(Runtimes[i].Get())) {
			CPURuntimes.Add(CPURuntime);
			UE_LOG(LogTemp, Warning, TEXT("CPU runtime available: %s"), *Runtimes[i]->GetRuntimeName());
		} else if (auto GPURuntime = Cast<INNERuntimeGPU>(Runtimes[i].Get())) {
			GPURuntimes.Add(GPURuntime);
			UE_LOG(LogTemp, Warning, TEXT("GPU runtime available: %s"), *Runtimes[i]->GetRuntimeName());
		} else if (auto RDGRuntime = Cast<INNERuntimeRDG>(Runtimes[i].Get())) {
			RDGRuntimes.Add(RDGRuntime);
			UE_LOG(LogTemp, Warning, TEXT("RDG runtime available: %s"), *Runtimes[i]->GetRuntimeName());
		} else {
			UE_LOG(LogTemp, Warning, TEXT("Non CPU/GPU/RDG runtime: %s"), *Runtimes[i]->GetRuntimeName());
		}
	}
}

becomes:

for (const auto& Name : GetAllRuntimeNames()) {
	UE_LOG(LogTemp, Warning, TEXT("Available runtime: %s"), *Name);
	TWeakInterfacePtr<INNERuntime> Runtime = GetRuntime(Name);
	if (Runtime.IsValid()) {
		if (auto CPURuntime = Cast<INNERuntimeCPU>(Runtime.Get())) {
			CPURuntimes.Add(CPURuntime);
			UE_LOG(LogTemp, Warning, TEXT("CPU runtime available: %s"), *Name);
		} else if (auto GPURuntime = Cast<INNERuntimeGPU>(Runtime.Get())) {
			GPURuntimes.Add(GPURuntime);
			UE_LOG(LogTemp, Warning, TEXT("GPU runtime available: %s"), *Name);
		} else if (auto RDGRuntime = Cast<INNERuntimeRDG>(Runtime.Get())) {
			RDGRuntimes.Add(RDGRuntime);
			UE_LOG(LogTemp, Warning, TEXT("RDG runtime available: %s"), *Name);
		} else {
			UE_LOG(LogTemp, Warning, TEXT("Non CPU/GPU/RDG runtime: %s"), *Name);
		}
	}
}

Then you will notice that APIs might return shared pointers instead of unique pointers and the way you create models isn’t uniform anymore, the method names contain the model type (for instance CreateModelGPU()).

Nicer error responses on some APIs: if (GPUModelInstance->SetInputTensorShapes(InputShapes) == EResultStatus::Fail)

Overall, nice cleanup and smooth upgrade.

gabaly92 · May 1, 2024, 4:26am

I am currently working with NNE ORT Beta in Unreal Engine 5.4, I am testing ORT CPU and ORT DML, so I tried a new model that has 1 float32 input and 3 float32 outputs and the same issue happens again, inference runs, I can post process the output and all is good then the engine crashes, the actor using the Neural Network basically just runs begin play and that’s it, it crashes after the code is executed, so looks like the issue is not with int64, I ran int64 models and Float16 models (they do still crash if you have more than one output though), all single output neural nets don’t crash the engine.

the exact same behavior happens in 5.3 as well with NNE Experimental, I am running 2 versions of the project now

unfortunately, I cannot share the project code here at this time, can you share your email, so I can share a link that has the project source code and a demo that shows the crash ? or let me know the best way that works for you to share the link

Thank you

gabaly92 · May 2, 2024, 12:10am

@mattai have you tried running inference with a Neural Net that has more than one output ?
I am talking in Unreal Engine 5.4 using NNE Beta, ORT CPU or ORT DML, not using the other experimental Runtimes ? if so, does it crash after running inference ? this is the behavior I am facing with all Multi-output neural nets, inference runs fine but it crashes after

mattai · May 2, 2024, 2:51am

The shape of my output is:

Outputs.SetNum(1);
Outputs[0].Shape = { static_cast<int32>(NumFrames) , 360 };
Outputs[0].Data.SetNum(NumFrames * 360);

ranierin · May 2, 2024, 9:07am

Hey @gabaly92

The team just checked and we have some models with multiple outputs in our testbench.

In addition I did a manual test to run a multi-output model from the web (you can find it here)

Also, I created and exported a simple model using torch:

import torch
import onnx

class MultiOutputTest(torch.nn.Module):
    def __init__(self):
        super(MultiOutputTest, self).__init__()
        self.l1 = torch.nn.Linear(1, 1)

    def forward(self, x):
        y = self.l1(x)
        z = x + y
        return y, z

model = MultiOutputTest()
torch.onnx.export(model, (torch.zeros(1)), 'MultiOutputTest.onnx')

Both models work without a crash. Maybe you can try if those models run on your side?
Please make sure that the memory of the input and the outputs remain alive (if you wrote blueprint functions, maybe just try to define the arrays as global statics and use pointers to the array to create the bindings).

Let us know how it goes!

gabaly92 · May 2, 2024, 8:32pm

I tried the model you attached, and as usual inference works fine but the engine still crashes after the end of begin play, attached is a demo of the crash of the model I am testing which is similar to the one you attached 1 float32 input and 3 float32 outputs, inference works fine, then it crashes, again this does not happen with single output neural nets, I am beginning to realize, it might be a C++ problem or I am misusing the Engine, here is a link to the demo and the code https://drive.google.com/drive/folders/1_HMVLjDjM9zHzjoiQuhjN-GKq88sQw8n?usp=sharing

can you please take a look at it and let me know what I need to change to prevent the crash ?
you will need to request access for the link

gabaly92 · May 2, 2024, 9:25pm

Your model is a single output model, so you need to setup only one output binding for it, for models like this I have no problem, the engine does not crash after running inference, the engine crashes with models like the one @ranierin mentioned, it has 2 outputs, so you need 2 output bindings for it, here is the model

models/validated/vision/body_analysis/ultraface/models/version-RFB-320.onnx at main · onnx/models (github.com)

mattai · May 2, 2024, 10:04pm

@gabaly92 Try maybe to run the editor in DebugGameEditor instead of DevelopmentEditor and see if you can get a breakpoint or a stack. I would have tried locally but your demo isn’t public (the gdrive link is private)

gabaly92 · May 2, 2024, 10:56pm

I tried debugging it for a really long time and unable to pinpoint which variable or pointer is causing the crash, there is so much going on, do you want to help debug it ? request access in the link

start from LandMarkDetection.cpp BeginPlay and work your way from there

also you will need to

Import the model to the project
Edit the paths in lines 138 and 155

let me know if you have any questions

gabaly92 · May 3, 2024, 8:13pm

@ranierin is it okay if you provide a sample C++ script running the model you referenced ? I was able to run inference with it but it is still crashing at the end of begin play, maybe I am not setting it up properly, I don’t know, my beginplay literally does two things, setup model and run inference, but as you saw in the demo and the logs, I setup the model, run inference, I get the right outputs and I am able to postprocess the outputs and get the right landmarks on the hand, it is just that it crashes after postprocess or at the end of begin play

if I run a single output model with the same setup, no crashes happen after inference and post processing

Thank you

gabaly92 · May 8, 2024, 7:06am

@ranierin @mattai I ran the debugger with engine symbols added to Unreal and I got a more detailed feedback on the error. again the code crashes at the end of beginplay after all the code executes successfully (load model, setup model, preprocess input, run inference, post process output), this only happens with a multiinput/multioutput neural net, any thoughts on the error ? here is the call stack

[Inlined] [UnrealEditor-Core.dll] _mi_page_malloc(mi_heap_s *, mi_page_s *, unsigned long long) 0x00007ffaedd7255c
[Inlined] [UnrealEditor-Core.dll] mi_heap_malloc_small(mi_heap_s *, unsigned long long) 0x00007ffaedd72559
[Inlined] [UnrealEditor-Core.dll] mi_heap_malloc(mi_heap_s *, unsigned long long) 0x00007ffaedd72559
[UnrealEditor-Core.dll] _mi_heap_malloc_zero(mi_heap_s *, unsigned long long, bool) 0x00007ffaedd72559
[UnrealEditor-Core.dll] mi_heap_malloc_zero_aligned_at(mi_heap_s *const, const unsigned long long, const unsigned long long, const unsigned long long, const bool) 0x00007ffaedd7995b
[UnrealEditor-Core.dll] mi_heap_realloc_zero_aligned_at(mi_heap_s *, void *, unsigned long long, unsigned long long, unsigned long long, bool) 0x00007ffaedd7a0b5
[Inlined] [UnrealEditor-Core.dll] mi_heap_realloc_zero_aligned(mi_heap_s *, void *, unsigned long long, unsigned long long, bool) 0x00007ffaedd7d2a3
[Inlined] [UnrealEditor-Core.dll] mi_heap_realloc_aligned(mi_heap_s *, void *, unsigned long long, unsigned long long) 0x00007ffaedd7d27b
[UnrealEditor-Core.dll] mi_realloc_aligned(void *, unsigned long long, unsigned long long) 0x00007ffaedd7d27b
[Inlined] [UnrealEditor-Core.dll] FMallocMimalloc::TryRealloc(void *, unsigned long long, unsigned int) MallocMimalloc.cpp:136
[UnrealEditor-Core.dll] FMallocMimalloc::Realloc(void *, unsigned long long, unsigned int) MallocMimalloc.cpp:155
[Inlined] [UnrealEditor-Core.dll] FLowLevelMemTracker::IsEnabled() LowLevelMemTracker.h:1086
[UnrealEditor-Core.dll] FMemory::Realloc(void *, unsigned long long, unsigned int) FMemory.inl:118
[UnrealEditor-Core.dll] TSizedHeapAllocator<32, FMemory>::ForAnyElementType::ResizeAllocation(int, int, unsigned long long, unsigned int) ContainerAllocationPolicies.h:725
[Inlined] [UnrealEditor-Engine.dll] TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> >::AllocatorResizeAllocation(int, int) Array.h:3064
[UnrealEditor-Engine.dll] TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> >::ResizeTo(int) Array.h:3141
[Inlined] [UnrealEditor-Engine.dll] TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> >::Empty(int) Array.h:1945
[Inlined] [UnrealEditor-Engine.dll] FCookStatsManager::CreateKeyValueArray(const wchar_t (&)[11], const wchar_t (&)[11], const wchar_t (&)[5], const wchar_t *&, const wchar_t (&)[10], const wchar_t *&&, const wchar_t (&)[6], long long &&, const wchar_t (&)[8], double &&, const wchar_t (&)[3], double &&, const wchar_t (&)[5], double &&, const wchar_t (&)[5], const FString &) CookStats.h:135
[UnrealEditor-Engine.dll] FCookStats::CallStats::LogStats'::2’::<lambda_1>::operator()(EHitOrMiss,bool) CookStats.h:252
[Inlined] [UnrealEditor-Engine.dll] FCookStats::CallStats::LogStats(TFunctionRef<void __cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)>, const FString &, const FString &, const wchar_t *) CookStats.h:271
[UnrealEditor-Engine.dll] FDerivedDataUsageStats::LogStats(TFunctionRef<void __cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)>, const FString &, const FString &) CookStats.h:432
[UnrealEditor-Engine.dll] ??R<lambda_13>@BodySetupCookStats@@QEBA@V?$TFunctionRef@$$A6AXAEBVFString@@AEBV?$TArray@U?$TKeyValuePair@VFString@@V1@@FCookStatsManager@@V?$TSizedDefaultAllocator@$0CA@@@@@@Z@@@Z(TFunctionRef<void __cdecl(FString const &,TArray<FCookStatsManager::TKeyValuePair<FString,FString>,TSizedDefaultAllocator<32> > const &)>) BodySetup.cpp:81
[Inlined] [UnrealEditor-Engine.dll] Invoke(BodySetupCookStats::<lambda_13> &, TFunctionRef<void __cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)> &&) Invoke.h:47
[Inlined] [UnrealEditor-Engine.dll] UE::Core::Private::Tuple::TTupleBase<TIntegerSequence >::ApplyAfter(BodySetupCookStats::<lambda_13> &, TFunctionRef<void __cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)> &&) Tuple.h:309
[UnrealEditor-Engine.dll] ?ExecuteIfSafe@?$TBaseFunctorDelegateInstance@$$A6AXV?$TFunctionRef@$$A6AXAEBVFString@@AEBV?$TArray@U?$TKeyValuePair@VFString@@V1@@FCookStatsManager@@V?$TSizedDefaultAllocator@$0CA@@@@@@Z@@@ZUFDefaultDelegateUserPolicy@@V<lambda_13>@BodySetupCookStats@@$$V@@UEBA_NV?$TFunctionRef@$$A6AXAEBVFString@@AEBV?$TArray@U?$TKeyValuePair@VFString@@V1@@FCookStatsManager@@V?$TSizedDefaultAllocator@$0CA@@@@@@Z@@@Z(TFunctionRef<void __cdecl(FString const &,TArray<FCookStatsManager::TKeyValuePair<FString,FString>,TSizedDefaultAllocator<32> > const &)>) DelegateInstancesImpl.h:868
[Inlined] [UnrealEditor-Core.dll] TMulticastDelegateBase::Broadcast(TFunctionRef<void __cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)>) MulticastDelegateBase.h:254
[UnrealEditor-Core.dll] TMulticastDelegate<void __cdecl(TFunctionRef<void cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)>), FDefaultDelegateUserPolicy>::Broadcast(TFunctionRef<void cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)>) DelegateSignatureImpl.inl:956
[UnrealEditor-Core.dll] FCookStatsManager::LogCookStats(TFunctionRef<void cdecl(const FString &, const TArray<FCookStatsManager::TKeyValuePair<FString, FString>, TSizedDefaultAllocator<32> > &)>) CookStats.cpp:32
[UnrealEditor-DerivedDataCache.dll] GatherDerivedDataCacheSummaryStats(FDerivedDataCacheSummaryStats &) DerivedDataCacheUsageStats.cpp:129
[UnrealEditor-StudioTelemetry.dll] FStudioTelemetryEditor::RecordEvent_Loading(const FString &, double, TArray<FAnalyticsEventAttribute, TSizedDefaultAllocator<32> >) StudioTelemetryEditor.cpp:180
[UnrealEditor-StudioTelemetry.dll] ??R<lambda_13>@?1??Initialize@FStudioTelemetryEditor@@QEAAXXZ@QEBA@PEAVUGameInstance@@@Z(UGameInstance *) StudioTelemetryEditor.cpp:685
[Inlined] [UnrealEditor-StudioTelemetry.dll] Invoke(<lambda_13> &, UGameInstance *&&) Invoke.h:47
[Inlined] [UnrealEditor-StudioTelemetry.dll] UE::Core::Private::Tuple::TTupleBase<TIntegerSequence >::ApplyAfter(<lambda_13> &, UGameInstance *&&) Tuple.h:309
[UnrealEditor-StudioTelemetry.dll] ?ExecuteIfSafe@?$TBaseFunctorDelegateInstance@$$A6AXPEAVUGameInstance@@@ZUFDefaultDelegateUserPolicy@@V<lambda_13>@?1??Initialize@FStudioTelemetryEditor@@QEAAXXZ@$$V@@UEBA_NPEAVUGameInstance@@@Z(UGameInstance *) DelegateInstancesImpl.h:868
[Inlined] [UnrealEditor-Engine.dll] TMulticastDelegateBase::Broadcast(UGameInstance *) MulticastDelegateBase.h:254
[UnrealEditor-Engine.dll] TMulticastDelegate<void cdecl(UGameInstance *), FDefaultDelegateUserPolicy>::Broadcast(UGameInstance *) DelegateSignatureImpl.inl:956
[UnrealEditor-Engine.dll] UGameInstance::StartPlayInEditorGameInstance(ULocalPlayer *, const FGameInstancePIEParameters &) GameInstance.cpp:572
[UnrealEditor-UnrealEd.dll] UEditorEngine::CreateInnerProcessPIEGameInstance(FRequestPlaySessionParams &, const FGameInstancePIEParameters &, int) PlayLevel.cpp:3141
[UnrealEditor-UnrealEd.dll] UEditorEngine::OnLoginPIEComplete_Deferred(int, bool, FString, FPieLoginStruct) PlayLevel.cpp:1589
[UnrealEditor-UnrealEd.dll] UEditorEngine::CreateNewPlayInEditorInstance(FRequestPlaySessionParams &, const bool, EPlayNetMode) PlayLevel.cpp:1853
[UnrealEditor-UnrealEd.dll] UEditorEngine::StartPlayInEditorSession(FRequestPlaySessionParams &) PlayLevel.cpp:2868
[UnrealEditor-UnrealEd.dll] UEditorEngine::StartQueuedPlaySessionRequestImpl() PlayLevel.cpp:1167
[UnrealEditor-UnrealEd.dll] UEditorEngine::StartQueuedPlaySessionRequest() PlayLevel.cpp:1064
[UnrealEditor-UnrealEd.dll] UEditorEngine::Tick(float, bool) EditorEngine.cpp:1901
[UnrealEditor-UnrealEd.dll] UUnrealEdEngine::Tick(float, bool) UnrealEdEngine.cpp:547
[UnrealEditor.exe] FEngineLoop::Tick() LaunchEngineLoop.cpp:5904
[Inlined] [UnrealEditor.exe] EngineTick() Launch.cpp:61
[UnrealEditor.exe] GuardedMain(const wchar_t *) Launch.cpp:182
[UnrealEditor.exe] LaunchWindowsStartup(HINSTANCE *, HINSTANCE *, char *, int, const wchar_t *) LaunchWindows.cpp:247
[UnrealEditor.exe] WinMain(HINSTANCE *, HINSTANCE *, char *, int) LaunchWindows.cpp:298
[Inlined] [UnrealEditor.exe] invoke_main() 0x00007ff773f7b95a
[UnrealEditor.exe] __scrt_common_main_seh() 0x00007ff773f7b939
[kernel32.dll] 0x00007ffb4aeb257d
[ntdll.dll] 0x00007ffb4b9caa48

the error is in line 136 of as shown in the image in the following engine file

C:\Program Files\Epic Games\UE_5.4\Engine\Source\Runtime\Core\Private\HAL\MallocMimalloc.cpp

here is my BeginPlay() that produced the output logs in the previous demo I posted:

void ULandMarkDetector::BeginPlay()
{
	Super::BeginPlay();

	UE_LOG(LogTemp, Warning, TEXT("##########-------------------- HandLandMarksDetection --------------------##########"));
	
	
	std::string imagePath = "C:/Users/ahmed/Documents/Unreal Projects/LandMarkDetection/Source/LandMarkDetection/hand_center_cropped.png";
    this->SourceImage = cv::imread(imagePath, cv::IMREAD_COLOR);
    
    // check if the image has been loaded correctly
    if(this->SourceImage.empty())
    {
        UE_LOG(LogTemp, Error, TEXT("Image cannot be loaded"));
    }
    else
    {
    	this->HandLandMarksInput = new cv::Mat(this->SourceImage);

    	this->LoadModel();
    	if (this->bModelLoaded)
    	{
    		UE_LOG(LogTemp, Display, TEXT("Model load sucess"));
    		this->SetupModel();
    		this->RunModel();
    	}
    }
}

this has been frustrating, I can’t figure it out, any feedback would be appreciated

Thank you

mattai · May 8, 2024, 7:22pm

Your stack trace seems to indicate that your game crashes before you even get to load your actor. Add a breakpoint inside your BeginPlay() and see if you can pinpoint the place where it crashes. It’s currently crashing when trying to allocate some memory when gathering stats about your cooked assets. You can try to clear your derived data, make sure your models were reimported when you upgraded UE and everything else is clean.

You might also want to use FString instead of std::string, not that it matters much, but you are probably going to want to refer to your Content folder down the line. (in general, it’s recommended to use UE’s data types when possible, especially when keeping data around like you do)

gabaly92 · May 9, 2024, 6:13am

Interesting, How is the actor not loaded, if the actor component runs it’s begin play, I mean LandMarkDetector is an actor component, the actor component has to be used on an actor that is valid, right ?

mattai · May 9, 2024, 4:04pm

I think the issue your are reporting is outside of the scope of this thread. Your logs don’t have anything NNE related. I’d suggest to try booting without your actor/model and see if that still works, if it does, re-add your actor, add a breakpoint at the beginning of BeginPlay and step through the execution until you get the crash. You should open a new thread though (feel free to tag me). Note that you seem to be on Mac, while most of us are on Windows.

ranierin · May 10, 2024, 6:13am

Sorry for the late reply, lot of things going on.

I agree with @mattai this does not seem to be NNE related. He also had some very good points:

Try to avoid using the std library but use the UE versions of it (FString, not std:string etc.)>
Also, try to set your project configuration to ‘Debug Editor’ instead of ‘Development Editor’ which could provide further insigts.
some debugger show the crash happening after a line of code while it actually happened inside, so maybe that was misleading.
Reimporting the models when switching versions is also a good idea to avoid any conflicts in versioning. A clean rebuild of the solution goes into the same direction

I hope you can figure it out!

gabaly92 · May 11, 2024, 4:10am

@mattai @ranierin I figured it out , I really appreciate the help. I spent a huge amount of time debugging this and I am so happy it paid off, I got both models working without a crash, finally:

LandMark model with 3 Concrete float32 inputs and 2 Concrete float32 outputs
Embedding model with 3 int64 inputs that are not concrete and 2 float32 outputs which are also not concrete

now I have a solution that is stable for the first time in a while, here are the changes that I made

Use Actor instead of Actor Component, for now this is apparently more stable
Move every variable that the model needs for preprocessing and postprocessing in the model manager
Move the model manager inside the actor class

I guess to the best of my knowledge and according to the demo, this guarantees that everything related to the Neural Net stays safe during play, this demo is using

Unreal Engine 5.4.1 and NNE Beta OrtCpu

@ranierin it would be nice if you can provide the following

The best practices when using the NNE Plugin
recommended modules for processing the neural network data given that it is actually always better use to Unreal Engine libraries as much as possible in favor of other 3rd party libraries, although 3rd party libraries are useful, OpenCV is really convenient for processing 2D images compared to Unreal Engine, also the opencv dnn module has some nice Machine learning related functions that are not available in Unreal, or most of them have to be written from scratch in Unreal Engine, for example NMS (Non Maximum Suppression)

Thank you

ranierin · May 14, 2024, 6:20am

Glad you figured it out! And thanks for sharing the solution, this will help others when running into the same issue.

Regarding best practices:
A 5.4 tutorial is still on our todo list, but best practices remain the same as with 5.3, we mainly changed internal stuff and some API naming inconsistencies.

Regarding 3rd party libraries:
Of course you can and should use them, but typically their are compiled outside the Engine and then linked with static/dynamic libraries. As long as you go this path, everything should be fine.