Full Body tracking for UE5 in real-time and 3D sensors support [Nuitrack][PLUGIN][FAB]

[Nuitrack]

Are you using Touch Designer? https://nuitrack.com/interactive-art

Information about the latest update

Download plugin:

Nuitrack - Real-time body tracking

Video about the new update:

How to make interactive particles in a simple way? @Orbbec3d Astra 2 sensor #pointcloud

Information about the latest update

If problems occur, read the Troubleshooting section carefully

We’re happy to present our real-time body tracking for Unreal Engine 5. It could be used without writing any code, simply connecting Nodes in Blueprints
It’s perfect for developing games , fitness apps and other interactive projects with body tracking.

Get started with examples showing different Nuitrack capabilities:

• Niagara - real-time interactive particles https://youtu.be/w7Mee0fhpg8?feature=shared
• Metahuman - animate metahuman avatar https://www.youtube.com/watch?v=mZu1i1h05UQ
• SkeletonJoints - check what the camera sees and how the skeleton joints are tracked
• AvatarMannequin - animate a game character using Nuitrack skeleton
• FittingRoom - how to create a virtual fitting room
• GesturesDemo - demonstrates how the gesture recognition works
• HandTracker - how to use hand tracking to interact with the UI

If you’re interested in Skeleton Tracking with Orbbec, Intel RealSense or Microsoft Xbox Kinect sensors (v1, v2, Azure) , you’re welcome to try https://www.unrealengine.com/marketplace/en-US/product/nuitrack-real-time-body-tracking

---

---

Tutorials :
How To Start - https://forums.unrealengine.com/t/real-time-full-body-tracking-for-ue5-nuitrack-plugin/1331349/9
How To Visualize Skeleton - https://forums.unrealengine.com/t/real-time-full-body-tracking-for-ue5-nuitrack-plugin/1331349/18
How to Animate Avatar (Text) - https://forums.unrealengine.com/t/update-real-time-full-body-tracking-for-ue5-nuitrack-plugin/1331349/24?u=nuitrack

How to make Interactive Particles (Niagara Point Cloud) - Interactive Niagara Particles with Kinect-like sensors [Nuitrack] | Community tutorial

---

---

Troubleshooting:

• Build Not Start\Crash
• Crash when level starts in Editor - try to launch UE with admin
• Where Samples and Blueprints?
• Poor tracking quality
  https://forums.unrealengine.com/t/real-time-full-body-tracking-for-ue5-nuitrack-plugin/1331349/19?u=nuitrack
• If you encounter startup errors, try deleting libmiddleware.dll, libnuitrack.dll and libopencv_core.dll from */Binaries/Win64.
  
  
• If blueprint is broken, try enabling plugins: Json Blueprint Utilities, Groom
• Error: “The program can’t start because *.dll is missing for your computer”. Try reinstalling the program to fix this problem". Try to copy *.dll file from “C:\Program Files\EpicGames\UE_5.X\Engine\Plugins\Marketplace\NuitrackPlugin\Source\NuitrackModule\lib\win64” to your build folder (next to the .exe file)

[Nuitrack]

You can also ask questions in our community and learn more about Nuitrack on github

Nuitrack Community https://community.nuitrack.com/
Github https://github.com/3DiVi/nuitrack-sdk

[Nuitrack]

Nuitrack has two skeleton tracking modes “Classical” and AI.
The “Classical” mode has a high speed of operation and can be run even on mobile devices:

Nuitrack | Another SDK vs Nuitrack SDK

[Nuitrack]

@Nuitrack

Nuitrack has two skeleton tracking modes “classical” and AI.
The classical one has a high speed of operation and can be run even on mobile devices

AI tracking uses the latest machine learning technologies , which allows you to better recognize complex poses. More sensitive to device performance than the “classical” algorithm
(Can also recognize some objects)

Nuitrack AI | Advanced skeleton tracking, face tracking and object detector

[Nuitrack]

At the moment, some NUITRACK functionality is not implemented in the plugin:

• face tracking (bbox, emotions, gender, age, years) [ ]
• object recognition
• multisensor tracking

So far, this can only be used through C++ code. Write to us if someone needs this functionality in the form of blueprint nodes

[Nuitrack]

At the moment there are several blueprints and levels for self-study. If the UE community has an interest in our plugin, we will definitely add tutorials and videos.

[PPPLab]

it is totally insane… i can’t wait…!! please give me a nuitrack tutorial for ue5!!!

[Nuitrack]

This week we plan to start publishing small video tutorials, subscribe to our channel

• How to start
• Blueprints and Levels Overview
• How to Visualize Skeleton
• How Animate Avatar (3D Model\Skeletal Mesh)
• …

[Nuitrack]

Nuitrack And Unreal. How to start
Don’t forget to subscribe to the channel and like this video

Nuitrack For Unreal Engine 5 (Blueprints) | How to Start

Coming soon: tutorial “How to Visualize Skeleton”

[Nuitrack]

[m_alitanveer]

Hey guys.
Great plugin. Sounds like something I can use for my project.

I have a couple of questions.

1. Can i use multiple Kinect sensors using this plugin? So technically I have 3 kinects connected to my system, and I want tracking data of closest/any tracked person from all 3 sensors. Possible?

2. Does this plugin support tracking via LIDAR? Can I track player movements / location via LIIDAR?

Blueprints are preferred but not a necessity, I just need these functionalities and I am perfectly fine with using C++ for these.

(Using this forum instead of your community to drive communication here)

[Nuitrack]

Hello @m_alitanveer

1. Nuitrack has support for working with multiple sensors at the same time. There are no necessary nodes in blueprint yet, but you can use it via C++. This example should help you https://github.com/3DiVi/nuitrack-sdk/blob/master/Examples/nuitrack_multisensor_sample/src/main.cpp#L67
   By the way, which kinects do you want to use (v1, v2, azure)?
2. Which sensor do you want to use? RealSense LiDAR L515 is working.

[MightyFely]

Could you make a tutorial how to use two kinect v2 to get a better tracking?

[MightyFely]

Would love to see the next steps please

[Nuitrack]

At the moment, this feature is under active development.
Stay tuned: https://github.com/3DiVi/nuitrack-sdk

• Holistic skeletal tracking with multiple sensors

[T0maytoe]

@Nuitrack

Cheers.

I have used your plugin in my project. It work correctly in my editor. But if I would like to package my project I can’t. First I got plugin modul error. I solved that problem, but now I absolutely cannot package the project. I got unknown error. Without the plugin I can. What do you think what could be the problem?

Thanks for you reply.

[Nuitrack]

Hello @T0maytoe

Can you tell me which version of Unreal you are using and provide any logs or screenshots?

This information may also help you
https://forums.unrealengine.com/t/real-time-full-body-tracking-for-ue5-nuitrack-plugin/1331349/20?u=nuitrack

[Nuitrack]

Hello. And here is a new video tutorial that will show how to work with joints and visualize the skeleton

[Nuitrack]

Each sensor has its advantages and disadvantages. Be sure to study the manufacturer’s documentation.
And you can study the general recommendations in this article
https://github.com/3DiVi/nuitrack-sdk/blob/master/doc/General_preparations.md

[Nuitrack]

Full Body tracking for UE5 in real-time and 3D sensors support [Nuitrack][PLUGIN][FAB]

[Mail] :bangbang:A new version of the beta-plugin has been released: Simplified Game Building. Now you don’t need to edit any files, just do the build. Removed errors from the output log that appeared at startup The size of the plugin has been significantly reduced New Version Beta Nuitrack UE5 plugin [Update] Real-time Full Body tracking for UE5 [Nuitrack][PLUGIN] I also remind you that tutorials have appeared on our channel (Subscribe Nuitrack Channel so as not to miss): …

It doesn’t seem to be relevant anymore. Now no additional actions are required to build

Deprecated Build Instruction

How to Build:

1. Open {YourProject}\Source\{YourProject}\{YourProject}.Build.cs
2. Add PrivateDependencyModuleNames.AddRange(new string[] { "NuitrackModule" });
3. “Platforms”->“Windows (for example)”->“Package Project”

There is one problem with the build now. The compiler complains about Try\Catch in the file NuitrackUtils.cpp in Update(). You can either delete this block yourself and then recompile it, or wait for the next plugin update.