Announcement

Collapse
No announcement yet.

[Plugin] RealSense

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    [Plugin] RealSense

    An Unoffical plugin for Intel's RealSense 3D camera. This is an early alpha release with limited support, expect an incomplete SDK bind.

    Currently supports hand tracking via joints, face tracking via landmarks, and gesture support. Other features are available in C++. Hopefully more of the RSSDK's features will be exposed to blueprint in the future

    Please note that RealSense requires Windows 8.1 and a 4th generation Intel Core Processor to function correctly, see the devkit page for details.

    Update: Use the official plugin, it is likely better: https://www.unrealengine.com/blog/in...nreal-engine-4

    Download
    Latest - 0.2.0 - UE4.9, RSSDK0.6

    Older
    UE4.8
    UE4.7
    UE4.5

    Resources
    Github - Main Plugin Source and Documentation
    Intel Plugin Thread - Intel Developer forum thread

    Installation
    1. Make sure you have the newest drivers
    2. Download Plugin
    3. Create new or open a project.
    4. Browse to your project (typically found at Documents/Unreal Project/{Your Project Root})
    5. Copy Plugins folder into your Project root.
    6. Copy Binaries folder into your Project root.
    7. Restart the Editor and open your project again.
    8. Select Window->Plugins. Click on Installed and you should see a category called Input and a plugin called RealSense Plugin now available.


    Quick Setup - How To Add a RealSense Component and RealSenseInterface to any Blueprint
    -Open the blueprint you wish to receive RealSense events.
    -Add the RealSense Component to your blueprint by clicking Add Component



    To receive events we now have to add an interface. Click on Class Settings and under Details find 'Add' under interfaces. Add the RealSenseInterface.



    Blueprint - Example Joint Debug Hands


    You can easily modify the graph to output the hands to a more realistic location by changing the added origin from actor origin to a hand offset component attached to the actor.

    Blueprint - Example Gestures
    To receive gestures you need to enable gestures before using them, e.g. in your begin play.



    Then simply subscribe to the event you are interested in. In the above example we output the body side appended to a verbose thumbs up print. This will show every time the gesture is detected.

    Do note that there is currently a performance penalty when you enable gesture detection, multi-threading gestures is on the to-do list.


    Changelist

    0.2.0
    -Multi-threading! alpha state though some things may not be safe
    -Supports non-locking joints and landmark streaming
    -Lots of UE RS classes added, not all available at this time
    -Update to UE4.9

    0.1.3
    -Drag and drop setup should now work without crashes with latest RSSDK (0.6)

    0.1.2
    -Update to UE4.8 and public release.
    Last edited by getnamo; 02-09-2016, 05:51 PM.
    Plugins: Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense

    #2
    is there anyway it could expose the image (depth etc) data as a texture?
    great plugin thanks

    Comment


      #3
      getnamo,

      Thanks for the plugin, I recently got the camera and hopefully I'll be able to check it out and perhaps contribute back to it.
      Linux Support page (Wiki): https://wiki.unrealengine.com/Linux_Support
      Linux Demos page (Wiki): https://wiki.unrealengine.com/Linux_Demos
      My Unreal Engine 4 Linux repo: https://github.com/3dluvr/UnrealEngine/

      Comment


        #4
        Originally posted by antinnit View Post
        is there anyway it could expose the image (depth etc) data as a texture?
        great plugin thanks
        This is the plan, just trying to find the time to dedicate towards extending functionality. It may be some time and I will always gladly accept pull requests from others if you want to push the plugin faster

        Originally posted by amigo View Post
        getnamo,

        Thanks for the plugin, I recently got the camera and hopefully I'll be able to check it out and perhaps contribute back to it.
        Awesome, extending the plugin should be fairly straight forward, the only thing holding me back currently is that some of the features seem to hit the FPS very hard and will probably require separate threads to work flawlessly. Looking forward to your pull requests!
        Plugins: Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense

        Comment


          #5
          Hi, I am getting the lighting solution rounded up then I am moving onto this at weekend. For my project the main feature is head tracking. How is this build as far as head tracking goes (6dop) and are there any pointers or essential reading docs that i should check out. Our secondary goal is voice commands, has anyone ventured into this?

          Thanks in advance
          My Marketplace Items

          My Website

          My Unreal Games

          Comment


            #6
            Hey, are you still developing this plugin? If yes, this was made for the new kits with R200 and F200 cameras?

            Thanks in advance!

            Comment


              #7
              Originally posted by rafaelbr View Post
              Hey, are you still developing this plugin? If yes, this was made for the new kits with R200 and F200 cameras?

              Thanks in advance!
              Still being developed, due to heavy cpu usage it's waiting for me to write a low latency multi-threaded architecture. Aiming to have something done this or next week.

              Personally developing on a F200 camera but the SDK should support all the RS devices.
              Plugins: Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense

              Comment


                #8
                Update to 0.2.0
                -Multi-threading! alpha state though some things may not be safe
                -Supports non-locking joints, landmark and gesture streaming
                -Lots of UE RS classes added, not all available for use at this time.
                -Update to UE4.9

                This is still an alpha plugin, but you should be able to use joints, landmarks and faces without impacting your game thread performance. Next up on the todo will be raw streams and polling support.

                NB: RealSense hits your cpu hard, quad core recommended.
                Last edited by getnamo; 10-28-2015, 10:53 PM.
                Plugins: Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense

                Comment


                  #9
                  Hi getnamo, I'm interested in using something like this to drive blend shapes on a facial rig. I've seen realsense being used for similar stuff on youtube using the SDK, do you think it's simple to implement something like this in your plugin? I suppose what I'm saying is does Intel have libs you can use for these purposes and can they be plugged into unreal? Well done on all the work so far by the way.

                  Comment


                    #10
                    Hey, getnamo. Great plugin! I noticed that Enable Gestures Graph is not available in Blueprint editor. (UFUNCTION with BlueprintCallable is commented out in RealSenseComponent.h). I think I can enable some of the gestures individuality for Enable All Gesture functionality is not available. Still need to do more digging to get the string for gesture types. You might want to include that info in your README to make it easier for people who want to use it. Hopefully, I don't find any other issues as I go along. Cheers!
                    http://www.mastersofconquest.com

                    Comment


                      #11
                      Also, are you able to provide the VS solution by chance?
                      http://www.mastersofconquest.com

                      Comment


                        #12
                        There is also some issue with the Real Sense Joint Get position world. It gives no data. Regards.
                        http://www.mastersofconquest.com

                        Comment


                          #13
                          Greetings Getnamo - I work for the Intel Corporation and want to chat with you

                          Greetings Getnamo - I work for the Intel Corporation and a dev on the RealSense SDK ... would like to chat with you about possible collaboration regarding your UE4 RealSense plugin. What is the best way i can reach you?

                          Comment


                            #14
                            It seems Intel has released their official plugin: https://www.unrealengine.com/blog/in...nreal-engine-4

                            I highly recommend future users use that version! Cheers for picking up the baton!
                            Plugins: Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense

                            Comment


                              #15
                              Has anyone had any luck building the RealSense plugin for 4.12? I can't get it to build :/

                              Comment

                              Working...
                              X