How to get libtorch working?

TheEath · November 9, 2021, 10:39am

Hi,
I’m trying to get libtorch working with UE4. I have some code written that uses libtorch but now I can’t get it to work within unreal. When i try to open the engine I get either obscure errors (“The game module “ue4_project” could not be loaded. There may be an operating system error, module may not properly set up, or a plugin which has been included into the build has not been turned on.”) or it crashes on the first time I try to use a function from libtorch.

I feel like I’m doing something wrong with loading the dll from libtorch.

Any help is appreciated.

Amethystium77 · December 1, 2021, 11:21am

Hi, I have only partially managed to run UE4 and Libtorch, but I still have a runtime error. In case it is of your help:

In order to replicate the progress:

Visual Studio 2019, Unreal Engine 4 (4.26.2) and Libtorch 1.10 (CPU and release version)

-Create a new Project in PyCharm
-Install PyTorch: pip3 install torch torchvision torchaudio
-Write the following code:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("D:\\resnet18.pt")

-Download Libtorch: Stable (1.10), Windows, LibTorch, C++/Java and CPU.
https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-1.10.0%2Bcpu.zip

-Extract the zip in your PATH.

-Create a project in Unreal Engine 4 named MyProject of type ThirdPerson and of type C++ (the default project is in mode Development Editor).

-Close the editor of UE4

-Open the properties in Project → MyProject Properties. Select the configuration Development_Editor and VC++ Directories.

-Add the paths in External Include Directories:

(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\include

(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\include\torch\csrc\api\include

-Add the library int Library Directories:

(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\lib

-Add PublicAdditionalLibraries int the MyProject.Build.cs:

public class MyProject : ModuleRules
{
	public MyProject(ReadOnlyTargetRules Target) : base(Target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
		PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "HeadMountedDisplay" });
		PublicAdditionalLibraries.Add("(PATH)\\libtorch-win-shared-with-deps-1.10.0+cpu\\libtorch\\lib\\torch_cpu.lib");
		PublicAdditionalLibraries.Add("(PATH)\\libtorch-win-shared-with-deps-1.10.0+cpu\\libtorch\\lib\\c10.lib");
		PublicAdditionalLibraries.Add("(PATH)\\libtorch-win-shared-with-deps-1.10.0+cpu\\libtorch\\lib\\torch.lib");
	}
}

-Paste the dynamic libraries (dlls) in the Win64 directory of the UE4 project: (PATH)\MyProject\Binaries\Win64
Librarys: asmjit.dll, c10.dll. fbgemm.dll, libiomp5md.dll, libiompstubs5md.dll, torch.dll, torch_cpu.dll and uv.dll

-Paste on the top of file MyProjectCharacter.h: #include <torch/script.h>. If you Build the project you get a lot of notes, warnings and errors.

-Replace : #include <torch/script.h> with:

#pragma warning( push )
#pragma warning(disable: 4582)
#pragma warning(disable: 4583)
#pragma warning(disable: 4458)
#pragma warning(disable: 4624)
#pragma warning(disable: 4800)
#pragma warning(disable: 4541)
#include <torch/script.h> // One-stop header.
#pragma warning( pop )

-Add the follow code in the constructor of your MyProjectCharacter.cpp

torch::device(torch::kCPU);
torch::jit::script::Module module = torch::jit::load("D:\\resnet18.pt");
assert(module != nullptr);
std::cout << "ok\n";

-Build the project, everything should run fine.

-Add the following code after the code above:

// Create a vector of inputs.
std::vector<torch::jit::IValue> inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));

// Execute the model and turn its output into a tensor.
at::Tensor output = module.forward(inputs).toTensor();

std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';

-Select Debug → Options and uncheck: Use the new Exception Helper

-Build and Run, then, you get an error in runt time in: at::Tensor output = module.forward(inputs).toTensor();

If you continue you get a new runtime error:

Unhandled exception at 0x00007FFE8618F199 (ntdll.dll) in UE4Editor.exe: 0xC0000374: A heap has been corrupted (parameters: 0x00007FFE861F77F0).

-If you continue your project your project will run normally. (for example: Question Answering: BERT(Bidirectional Encoder Representation with Transformers) with LibTorch & UE4 - YouTube)

-I have tried for a couple of weeks to resolve the error at runtime (for example, disabling optimization with #pragma optimize ("", off)), but I have not succeeded. The error in the heap continues.

Amethystium77 · December 1, 2021, 11:22am

I have managed to partially integrate Libtorch with unreal engine 4, but I have a runtime error.

I have used: Visual Studio 2019, Unreal Engine 4 (4.26.2) and Libtorch 1.10

In order to replicate the progrees:

Create a new Project in PyCharm
Install PyTorch: pip3 install torch torchvision torchaudio

Write the following code:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("D:\\resnet18.pt")

Download Libtorch: Stable (1.10), Windows, LibTorch, C++/Java and CPU.
https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-1.10.0%2Bcpu.zip
Extract the zip in your PATH.
Create a project in Unreal Engine 4 named MyProject of type ThirdPerson and of type C++ (the default project is in mode Development Editor).
Close the editor of UE4
Open the properties in Project → MyProject Properties. Select the configuration Development_Editor and VC++ Directories.
Add the paths in External Include Directories:
(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\include,
(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\include\torch\csrc\api\include
Add the library int Library Directories:
(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\lib
Add PublicAdditionalLibraries int the MyProject.Build.cs:

//code

public class MyProject : ModuleRules
{
public MyProject(ReadOnlyTargetRules Target) : base(Target)
{
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
PublicDependencyModuleNames.AddRange(new string[] { “Core”, “CoreUObject”, “Engine”, “InputCore”, “HeadMountedDisplay” });
PublicAdditionalLibraries.Add("(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\lib\torch_cpu.lib");
PublicAdditionalLibraries.Add("(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\lib\c10.lib");
PublicAdditionalLibraries.Add("(PATH)\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\lib\torch.lib");
}
}
Paste the dynamic libraries (dlls) in the Win64 directory of the UE4 project: (PATH)\MyProject\Binaries\Win64
Librarys: asmjit.dll, c10.dll. fbgemm.dll, libiomp5md.dll, libiompstubs5md.dll, torch.dll, torch_cpu.dll and uv.dll
Paste on the top of file MyProjectCharacter.h: #include <torch/script.h>. If you Build the project you get a lot of notes, warnings and errors.
Replace : #include <torch/script.h> with:

//code

#pragma warning( push )
#pragma warning(disable: 4582)
#pragma warning(disable: 4583)
#pragma warning(disable: 4458)
#pragma warning(disable: 4624)
#pragma warning(disable: 4800)
#pragma warning(disable: 4541)
#include <torch/script.h> // One-stop header.
#pragma warning( pop )

Add the follow code in the constructor of your MyProjectCharacter.cpp

//code

torch::device(torch::kCPU);
torch::jit::script::Module module = torch::jit::load(“D:\resnet18.pt”);
assert(module != nullptr);
std::cout << “ok\n”;
Build and Run the project, everything should run fine.
Add the following code after the code above:

//code

// Create a vector of inputs.
std::vectortorch::jit::IValue inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));

// Execute the model and turn its output into a tensor.
at::Tensor output = module.forward(inputs).toTensor();

std::cout << output.slice(/dim=/1, /start=/0, /end=/5) << ‘\n’;
Select Debug → Options and uncheck: Use the new Exception Helper
Build and Run, then, you get an dialog box with the error in runt time in: at::Tensor output = module.forward(inputs).toTensor();
If you press Continue you get a new runtime error:
Unhandled exception at 0x00007FFE8618F199 (ntdll.dll) in UE4Editor.exe: 0xC0000374: A heap has been corrupted (parameters: 0x00007FFE861F77F0).
If you press Continue your project will run normally (for example: Question Answering: BERT(Bidirectional Encoder Representation with Transformers) with LibTorch & UE4 - YouTube).

I have tried for a couple of weeks to resolve the error at runtime (for example, disabling optimization with #pragma optimize ("", off)), but I have not succeeded. The error in the heap continues.