Download

[Plugin] RealSense

An Unoffical plugin for Intel’s RealSense 3D camera. This is an early alpha release with limited support, expect an incomplete SDK bind.

Currently supports hand tracking via joints, face tracking via landmarks, and gesture support. Other features are available in C++. Hopefully more of the RSSDK’s features will be exposed to blueprint in the future

Please note that RealSense requires Windows 8.1 and a 4th generation Intel Core Processor to function correctly, see the devkit page for details.

Update: Use the official plugin, it is likely better: See The World In New Ways With The Intel RealSense Plugin For Unreal Engine 4 - Unreal Engine

Download
Latest - 0.2.0 - UE4.9, RSSDK0.6

Older
UE4.8
UE4.7
UE4.5

Resources
Github - Main Plugin Source and Documentation
Intel Plugin Thread - Intel Developer forum thread

Installation

  1. Make sure you have the newest drivers
  2. Download Plugin
  3. Create new or open a project.
  4. Browse to your project (typically found at Documents/Unreal Project/{Your Project Root})
  5. Copy *Plugins *folder into your Project root.
  6. Copy Binaries folder into your Project root.
  7. Restart the Editor and open your project again.
  8. Select Window->Plugins. Click on Installed and you should see a category called Input and a plugin called RealSense Plugin now available.

Quick Setup - How To Add a RealSense Component and RealSenseInterface to any Blueprint
-Open the blueprint you wish to receive RealSense events.
-Add the RealSense Component to your blueprint by clicking Add Component

To receive events we now have to add an interface. Click on Class Settings and under Details find ‘Add’ under interfaces. Add the RealSenseInterface.

Blueprint - Example Joint Debug Hands

You can easily modify the graph to output the hands to a more realistic location by changing the added origin from actor origin to a hand offset component attached to the actor.

Blueprint - Example Gestures
To receive gestures you need to enable gestures before using them, e.g. in your begin play.

Then simply subscribe to the event you are interested in. In the above example we output the body side appended to a verbose thumbs up print. This will show every time the gesture is detected.

Do note that there is currently a performance penalty when you enable gesture detection, multi-threading gestures is on the to-do list.

Changelist

0.2.0
-Multi-threading! alpha state though some things may not be safe
-Supports non-locking joints and landmark streaming
-Lots of UE RS classes added, not all available at this time
-Update to UE4.9

0.1.3
-Drag and drop setup should now work without crashes with latest RSSDK (0.6)

0.1.2
-Update to UE4.8 and public release.

is there anyway it could expose the image (depth etc) data as a texture?
great plugin thanks

getnamo,

Thanks for the plugin, I recently got the camera and hopefully I’ll be able to check it out and perhaps contribute back to it.

This is the plan, just trying to find the time to dedicate towards extending functionality. It may be some time and I will always gladly accept pull requests from others if you want to push the plugin faster :slight_smile:

Awesome, extending the plugin should be fairly straight forward, the only thing holding me back currently is that some of the features seem to hit the FPS very hard and will probably require separate threads to work flawlessly. Looking forward to your pull requests!

Hi, I am getting the lighting solution rounded up then I am moving onto this at weekend. For my project the main feature is head tracking. How is this build as far as head tracking goes (6dop) and are there any pointers or essential reading docs that i should check out. Our secondary goal is voice commands, has anyone ventured into this?

Thanks in advance :slight_smile:

Hey, are you still developing this plugin? If yes, this was made for the new kits with R200 and F200 cameras?

Thanks in advance!

Still being developed, due to heavy cpu usage it’s waiting for me to write a low latency multi-threaded architecture. Aiming to have something done this or next week.

Personally developing on a F200 camera but the SDK should support all the RS devices.

Update to 0.2.0
-Multi-threading! alpha state though some things may not be safe
-Supports non-locking joints, landmark and gesture streaming
-Lots of UE RS classes added, not all available for use at this time.
-Update to UE4.9

This is still an alpha plugin, but you should be able to use joints, landmarks and faces without impacting your game thread performance. Next up on the todo will be raw streams and polling support.

NB: RealSense hits your cpu hard, quad core recommended.

Hi getnamo, I’m interested in using something like this to drive blend shapes on a facial rig. I’ve seen realsense being used for similar stuff on youtube using the SDK, do you think it’s simple to implement something like this in your plugin? I suppose what I’m saying is does Intel have libs you can use for these purposes and can they be plugged into unreal? Well done on all the work so far by the way.

Hey, getnamo. Great plugin! I noticed that Enable Gestures Graph is not available in Blueprint editor. (UFUNCTION with BlueprintCallable is commented out in RealSenseComponent.h). I think I can enable some of the gestures individuality for Enable All Gesture functionality is not available. Still need to do more digging to get the string for gesture types. You might want to include that info in your README to make it easier for people who want to use it. Hopefully, I don’t find any other issues as I go along. Cheers!

Also, are you able to provide the VS solution by chance?

There is also some issue with the Real Sense Joint Get position world. It gives no data. Regards.

Greetings Getnamo - I work for the Intel Corporation and want to chat with you

Greetings Getnamo - I work for the Intel Corporation and a dev on the RealSense SDK … would like to chat with you about possible collaboration regarding your UE4 RealSense plugin. What is the best way i can reach you?

It seems Intel has released their official plugin: See The World In New Ways With The Intel RealSense Plugin For Unreal Engine 4 - Unreal Engine

I highly recommend future users use that version! Cheers for picking up the baton!

Has anyone had any luck building the RealSense plugin for 4.12? I can’t get it to build :confused:

Since 4.10 I recommend using the official realsense plugin found here: GitHub - GameTechDev/UE4RealSensePlugin: This UE4 plugin provides support for the Intel RealSense SDK as noted in the last reply, if that one fails to compile, add a github issue to the repo.

That’s the one I’ve been trying to build but it says something about missing PCX3DSCAN.H
Just wondering if anyone has a working version… Also not sure if handtracking is already implemented in the official plugin.

Hello, I need real sense to my projects, and I build new version of plugin for 4.19. Maybe it could be interesting for you, here is the link [PLUGIN] Intel Real Sense plugin 4.19 with intel API v2 2016 - Work in Progress - Unreal Engine Forums