I want to use yolov7 on unreal engine

[seozero00]

I am a college student studying UAV system in Korea.
I recently encountered Unreal Engine and I want to use yolo here, but I don’t know how
how to do it?

thanks

[SupportiveEntity]

Hey there @seozero00! Neural Networks are pretty far out of my wheelhouse and I couldn’t find solid resources on integrating YOLO specifically, are you intended to just render out training images or integrate it into the engine for real time use?

[seozero00]

@SupportiveEntity
I would like to detect buildings (landmarks) within a virtual environment in real-time using a camera. I believe building and invoking YOLO from a C++ class in an Unreal Engine project is the way to go, but I am struggling with the process as it is unfamiliar to me.

Thank you.

[SupportiveEntity]

Understood. There is an API made for Neural Network purposes inside the engine that might be of use to you, though it’s usually used a bit differently from what I gather. Since it can use ONNX runtimes it may be applicable, though you’ll have to vet that claim as I’m mostly a layman to ML. Nico may know of better resources however!

Course: Neural Network Engine (NNE)

In this course you will learn everything needed to empower your game with AI using Unreal Engine’s neural network engine NNE. https://dev.epicgames.com/community/learning/courses/e7w/unreal-engine-neural-network-engine-nne

Disclaimer: One or more of these links are unaffiliated with Epic Games. Epic Games is not liable for anything that may occur outside of this Unreal Engine domain. Please exercise your best judgment when following links outside of the forums.

However I couldn’t find any resources on YOLO implementation with Unreal, it does seem that some users have done it, though their repo is undisclosed.

Live YOLO Object Detection on Unreal Engine Scene - Demo

Here’s a Repo using YOLOv7 in C++ that could offer some inspiration, though it’s not built for Unreal Engine.

GitHub - ShahriarRezghi/yolov7-cxx

[ranierin (Nico)]

Thanks for looping me in @SupportiveEntity

Hey @seozero00
Yes, you can use NNE for this purpose. There are several ways of doing this, with increasing difficulty. I would recommend to start easy and then make it more performant/difficult as you go.

If you have your YOLO in onnx format, I would just start with NNERuntimeORTCpu. It wont be super efficient, but easier to start with, get used to NNE and also for debugging. You can follow above linked course to set everyting up. In the short:

• enable the right plugins
• drag and drop your .onnx model into the editor
• get the runtime, load the model-asset and create a model instance
• get your image on cpu and feed it into the neural network to get your bounding boxes

After managing to get proper results, you can then try to switch from a cpu runtime to a gpu runtime: You still feed the image from CPU, but the neural network evaluation happens on the GPU and thus you will probably see a huge speedup.

The last and most difficult is then to use an RDG interface: you feed your camera image directly on the GPU and thus avoid any syncronisation with the CPU.

I hope that helps, let us know how it goes