[Plugin] Leap Motion - Event Driven

The official Leap Motion plugin for the Unreal Engine 4.

You can use convenience blueprints to just select and play or use a custom approach via blueprints or C++. See relevant sections in https://.com/leapmotion/LeapUnreal for details.

NB: Only supports Windows 64 & 32

Download

https://img.shields.io//release/leapmotion/leapunreal.svg

Latest: https://.com/leapmotion/LeapUnreal/releases

Older
Engine (4.11-4.18) - It’s already in the engine, just enable and restart!
0.9.9 for UE4.10 / Leap SDK 2.3.1 - Mac Binaries synced to 4.10

Documentation & Support
Leap Plugin Documentation - https://.com/leapmotion/leapunreal
Examples - https://.com/leapmotion/leapunrealmodules
Anything not covered in the documentation or examples is best asked at https://forums.leapmotion.com/

Resources
- Main Plugin Source and Documentation
Leap Motion C API documentation - For C++ programmers who want finer control
Leap Community UE4 Plugin Thread - Old leap motion forum thread for v2 plugin

**How to Use

https://camo.githubusercontent.com/607e36569294aa047afa4d1c67d0e27488a1861c/687474703a2f2f692e696d6775722e636f6d2f454154727953512e676966
**

See https://.com/leapmotion/LeapUnreal

*Changelist
Latest changes available at: https://.com/leapmotion/LeapUnreal/releases

3.0.0
-* Switching off leap service should no longer crash on exit

  • Ensure project plugin overrides old LeapC.dll
  • Updates and fixes for epic engine code
  • Switched to Async module for threading
  • Update internal API string to 4.0
  • Track motion controllers in bodystate, add custom device tag filters for selective animation consumption
  • Add support for Leap Image API from v4 SDK, events available from leap component. Requires setting appropriate policy
  • Add fix for #4](https://.com/leapmotion/LeapUnreal/issues/4) using scope locks

2.17.1

  • First LeapUnreal release
  • Updated Leap SDK to v4.0 with improved tracking

2.15

  • Compile fixes for 4.19. Uses new IXRTrackingSystem
  • Now works as both engine and project plugin. Will auto-detect environment and adjust dependencies.
  • Engine plugin requires IWYU only. All includes changed to this preference.
  • BS Confidence value fixes for 4.0 sdk, will work for 3.2 as well
  • misc fixes and doc changes

2.12

  • Added BodyStateBone Component. Similar to Motion Controller components, but it will track any body state bone from a drop down list.
  • Added BodyStateEstimator Component, which allows merging and updating skeleton data in a separate component which gets fed into the final merged skeleton. Useful for estimating body positions from incomplete data.
  • Fixed position offsets not being overwritten when using default tracking modes

2.11.2

  • Fixed compile warning for packaged builds
  • added log forwarding from leap service
  • enabled hot-restarting the leap service
  • added more verbose logging for problem states

2.11.0

  • Fixed and added auto-rigging for much wider mesh setups. See https://.com//leap-ue4-…custom-rigging for documentation.
  • Exposed global position and rotation offsets used to bring alignment 1:1
  • Fixed #28
  • Fixed #31
  • Added #30
  • Fixed #29

2.10.0

  • Compile fixes for 4.18

2.9.3

  • Compile fixes for 4.17
  • Fixed bodystate for packaged game
  • Added finger extension state to BSFinger

2.9.0

  • Default rigging set to bodystate low poly
  • significant changes to body state anim graph collapsing all blueprint code into a single node using a mapped data structure
  • body state skeleton auto-set on animation start
  • first pass at auto-mapping arbitrary bones to bodystate bones.
  • editor friendly method to adjust auto-mapped results in defaults, allowing for easy fine tuning
  • hand rigging now deforms to match your tracked data
  • time warp updates to sync with newly deformed hands due to now using bone positions

2.8.0

  • Renamed module from LeapMotionC to LeapMotion, issue #19
  • Misc body state offset fixes to sync behavior with low poly direct rigging

2.7.0

  • Compile fixes for 4.16
  • Compile fixes for #16
  • Updated documentation

2.6.7

  • Merged as b46ad8d
  • Oculus Timewarp fixes, the platform should have good and stable timewarp default settings
  • Three mode tracking optional settings allowing the choice of whether you want low latency, smooth, or balanced tracking
  • All Leap settings and options as a FLeapOptions struct for more finer grained control available globally
  • Added FLeapStats for polling information about frame extrapolation and device details, also available globally

2.6.1

  • Working timewarp implementation
  • Leap frame interpolation and extrapolation added
  • Default settings to extrapolate to current gamethread time + 5.5ms TW
  • More optimization settings have been exposed
  • Profiling cycle counters added
  • Fixes and tweaks

2.4.2
-First Pre-Release with proper packaging support.

2.0 - 2.4
-Internal development builds

2.0.0
-Integrated directly in 4.11 engine
-Added Image Hands
-Added Component Hands
-Thorough code commenting
-Automatic packaged support
-Added convenience LeapFloatingHandsCharacter

0.9.9
-Updated Mac binary for 4.10

0.9.8
-Updated binaries to UE4.10

0.9.7
-Fixed packaging for 4.9.2
-Added EnableBackgroundTracking toggle on LeapController allowing you to
track when the window isn’t focused
-Updated Echo hands to hide when not being tracked, idle
animations for Echo hands have been removed. Use
LeapEchoHandsRiggedCharacter for a rigged mesh hand without attached
body, use the collision variant for physics enabled version.

0.9.6
-Binaries synced with UE 4.9
-Leap SDK synced with 2.3.1
-Added Echo convenience hands (rigged floating hands)

0.9.5
-Updated binaries to UE4.8

0.9.4
-Updated Leap SDK to 2.2.6

0.9.3
-Added LeapCollisionCharacter which adds physics collision enabled by
default. This can be enabled for other setups by using the PhysicsActor
collision preset.
-Updated PhysicsAsset used for collision
-Removed old content
-Fixed SphereRadius property link found on LeapHand

0.9.2
-Mac Binaries compiled for UE 4.7
-Leap SDK, libs and dlls updated for SDK 2.2.4

0.9.1
-Cleaned up inheritence structure, LeapAnimBodyActor renamed to
LeapAnimBodyConnector. Added Passthrough sub-class
-LeapPassthroughCharacter now contains the subclass. Rigged characters
contain no passthrough code
-Fixed elbow rotation bug

0.9
-Added properly HMD warped image passthrough support through both
materials and a convenience character (LeapPassthrougCharacter)
-128bit texture support resolved for Distortion textures. Use Distortion
for proper float data, use DistortioUE to visualize the texture for
debug purposes.
-Optional content moved into plugin, delete if you don’t use it and wish
to save space
-Rigging update to be more loosely coupled. Character anim blueprint
only depends on AnimBody obtained through a simple and safe
AnimBodyInterface
-Rigged character finger rotation bug fixed
-Updated leap bind to 2.2.3, much improved tracking gained.
-Added R->Reset position to rigged characters, change key bind to
reflect own bindings

0.8.3
-Now supports Mac OSX
-Fixed Finger->Hand crash on exit
-Update to Leap SDK 2.2.1

0.8.2
-Now supports UE 4.6

0.8.1
-Fixed Memory leaks which would cause crash around 60second mark (GC
interval) for list-based objects using extended and append
-Added appendTools and appendFingers to pointableList
-LeapFrame got its API updated to include all of the missing functions,
should now reflect the leap api nearly completely (except 64bit values
in blueprint that are not supported)
-Added log entry with plugin version at game start for easier debugging

0.8.0
-Basic Rigged leap character added. Works like the LeapDebugCharacter,
change your player controller to VRPlayercontroller and pawn to
LeapRiggedCharacter
-Updated bind to Leap SDK 2.2.0
-Ensured consistency by prepending leap to all classes, this will likely
break compatibility with old blueprints.
-Fixed dynamic object allocation architecture
-Added basic AnimBody, Hand, Finger, and Bone for convenient animation
state and basis conversion.

0.7.11
-You can now check which finger you have by dragging out the Type
property from Finger.

0.7.10
-Added a convenience Leap Debug Hand actor, allowing for easy display of
debug hands
-Added a convenience Leap Debug Character which is set up to show debug
hands in both HMD and Default mode. Simply toggle your hmd (alt+enter)
to test HMD mode.

0.7.9
-added Input Mapping support for pinch grab and palm orientation for
each hand. See documentation for details
-added PalmOrientation convenience function to Leap Hand. Origin is
defined as palm flat facing down, fingers pointing away.

0.7.8
-Update to Leap SDK 2.1.6. Includes changes from leap
(https://developer.leapmotion.com/doc…#version-2-1-6)

0.7.7
-If you tick auto-shift on hmd, this will automatically add the
eye->leapmount offset (8cm default value). Your vr hands should now be
completely 1:1
-Added Hand Type enum to Hand and relevant gestures. You can now easily
switch the hand type instead of checking isLeft/Right to reduce bp
clutter.
-Added Leap Plugin Direction (Cardinal Direction) to gestures which
emit directions (Screen Tap, Key Tap, and Swipe). This is a convenience
property which will allow you to easily switch the property to get basic
swipe left/right/up/down/towards/away detection. If you need a more
precise direction, use the direction vector.

0.7.6
-Added Raw Image events which give much cleaner support and memory
efficient toggling of streaming images from the device.
-Multiple components are now properly supported. Reason for error was
mismanaged GC of on-demand UObject subclasses. Plugin will still emit
multi-component warning for performance reasons (redundancy/duplication).
-All logging switched to LeapPluginLog sub-category

0.7.5
-At the moment multiple components cause a crash after 40-60sec of
detecting a hand/finger, until this error is resolved, the plugin will
output a warning to your logs if you add more than one total leap
component

0.7.4
-Normalized vectors such as palm normal were not properly adjusted for
HMD view direction, this is has now been fixed.

0.7.3
-palm normal scaling fixed.

0.7.2
-Circle gesture radius now in UE scale
-Swipe Gesture speed now in UE scale

0.7.1
-Hidden Leap header and private implementation from UClasses
-Fixed image stability, will now correctly resize when requested.
-R8 texture no longer causes memory leaks
-Added Gesture sub-classes and corresponding events (e.g. Swipe Gesture
Detected). Will allow for easy determination of direction and other
properties of gestures. You can also cast general gesture detection to
the correct sub-class gesture.
-Added own category of logging, pay attention to log output to help you
with debugging errors.

0.7
-Added full support for Image API! see post or readme for an example use case.
-Fixed a lot of memory leaks, should have no more memory leaks and purr like a kitten.

0.6.4
-Fixed lack of categorization caused by previous updates in some of the blueprints classes
-Cleaned up a lot of the blueprint functions, turning all the relevant
ones into properties. This will reduce the number of execution pins.
-Things which return a new object generally remain a function for
potential performance reasons

0.6.3
-Changed plugin to runtime to support shipping builds
-See the readme for shipping/packaging instructions.

0.6.2
-Updated to UE4.5
-Removed the need to SetInterfaceDelegate! Just adding the Component and
adding the LeapEventInterface is enough.
-Added Image policy flag support, actual image handling is still not
working at this time. You can query about image dimension and other
stats, just not the raw image.

0.6.1
-OptimizeForHMD now contains 2 more booleans, one for auto rotation
adjustment and one for auto position adjustment. Setting all three to
enabled will make your reported leap positions automatically adjust for
your head position and look direction. VR easy mode enabled!

0.6
-Setting policy to optimize for hmd will automatically rotate position
data
-Fixed pinch detection stability, grabbing will now override pinching
event
-Changed event names to better reflect their purpose
-Added gesture event, set gesture support at startup to receive this
event
-Added polling data for: interaction box, pointables, gestures, images
(basic support)

0.5.2
-Added Event Driven architecture, Add LeapEventInterface to your
blueprint class and call LeapController->SetDelegate(self). Your class
will now receive leap events.
-Added Arm support, please see official leap documentation for reference
-Plugin now automatically converts leap space position data to UE space
data including scale (mm->cm)
-Added support for HMD Optimized tracking (also known as top down
tracking). Set OptimizeForHMD(true) on your LeapController.

0.5.1
-Updated to Leap SDK 2.1.5+22693
-Enabled getting finger data. Use Hand->Fingers

0.5.0
-compile fixes and updated to UE4.4

Hey man, came from the Leap forums! I like this, perfect for what I want to do. Am going to give it a try and see what I can do with it :slight_smile:

Your documentation and explanation is awesome by the way!

Hello. I have a question.

This did not work for OptimizeForHMD set true.

Will need to have the OVR?

Updated plugin to v0.6 with the following changes:

-Setting policy to optimize for hmd will automatically rotate position
data
-Fixed pinch detection stability, grabbing will now override pinching
event
-Changed event names to better reflect their purpose
-Added gesture event, set gesture support at startup to receive
gesture events (refer to leap api)
-Added polling data for: interaction box, pointables, gestures, images
(basic support)

This was working according to the leap api, it expected you to do rotations manually.

Since this is the expected use case, I’ve added automatic rotation changes when you enable optimize for HMD, try again and let me know if that is what you wanted.

Edit: 0.6 only adjusted for front facing rotation, 0.6.1 added HMD orientation and position adjustments as additional options for OptimizeForHMD.

First off thanks for the contribution! Hopefully between what you have going on here and what Leap has going on in their forums we’ll eventually get an official LeapMotion plugin for UE4.

Anyways, I am also developing for the Oculus Rift, and using the above blueprint event graph layout with OptmizeForHMD checked. The debug boxes only draw in any meaningful relation to my character when I look straight up at the ceiling where the Leap is in it’s Non-HMD orientation. This could be tied to the fact that I’m using a first person template and not third person as you were, and may require some additional rotations/translations to place the debug spheres properly in relation to the headset.

Note sure if it’s relevant, but Leap (Ultraleap for Developers) says, “starting with v2.1.3, you must set the policy flags POLICY_IMAGES and POLICY_OPTIMIZE_HMD to access the Image API and head-mounted tracking in your application”.

Hoping to have the time in the near future to start digging through their VRIntro source code.

Updated plugin to 0.6.1 which contains convenience options for VRMode.

You will now see autoRotation and autoShift booleans in addition to useTopdown for OptimizeForHMD. Setting all three to enabled will make your reported leap positions automatically adjust for your head position and look direction.

VR easy mode enabled!

This is what OptimizeforHMD does. Image API is currently the only real feature not properly supported (may get to it at some point).

Try the auto-rotation and auto-shift options and let me know if that fixes things! If you’re using the VR mode I suggest you shrink the debug spheres radius and scale to 1 as this will give that 1:1 feel.

Thank you so much for this! Finally an easy way to experiment with the leap and vr in ue4.

The raw image support would be interesting for me only in the sense that it would enable a somewhat easy port of this which is essential for a tracking thing I’m trying. (opentrack use in ue4 is somewhat problematic for me)

Update to 0.6.2
-Updated to UE4.5
-Removed the need to SetInterfaceDelegate! Just adding the Component and
adding the LeapEventInterface is enough.
-Added Image policy flag support, actual image handling is still not
working at this time. You can query about image dimension and other
stats, just not the raw image.

Updated documentation to reflect newer options and added some documentation regarding gestures. Full image support is missing at this time.

Hi,
and thanks for sharing this plugin with us :).
However… I’m currently having a problem with the plugin… as soon as I add the interface to my controller (MyController) and launches the project I end up with a message saying “Fatal error!” and that the game crashed and the message that’s displayed refers to a bunch of addresses and paths (as far as I can tell all the paths do exist).

I have added the Bin and Plugin folders and all the files to my project, and the plugin is enabled.

Im using the latest Leap SDK and UE4.4.3, I have also confirmed that my leap is working with other leap projects.
Im not sure if it matters but Im using Windows 8.1 x64… if that’s helpful in some way :).

Any ideas?

Hi !! First I want to thank you very much for your plugin :slight_smile: I was waiting a lot for that !! :smiley:
I was wondering if you could explain me if there is a way to allow the hands to follow the same rotation of the character Root (Capsule Component) ? ^^
When i am rotating my character, the hand doesn’t rotate :’
Thank you ! :wink:

Ok… so I just tried the updated version of the plugin with UE4.5 and also changed my projects directory to the default path… but now I keep getting this error:


Any ideas?

If its possible it would be really nice to have a sample project to test with to make sure that I have setup everything right :slight_smile:

Just did a blank project (no code) test with the currently uploaded 0.6.2 plugin from this page ( download), it worked without problems.

Please ensure you’re installing this correctly


Also check that you’re using the Windows x64 platform (XP/Vista/7/8 x64), I have not tested this plugin for any other platform and will need to add relevant binaries before it will work. If both of those cases are correct and it is still not working, send me your logs (found in {ProjectRoot}/Saved/Logs)

You have to rotate the positions you get from your leap by your control rotation like so


Make sure the order you do this is correct as it will rotate the vector around its origin. In most cases this will be before you convert it to world space.

Thank you very much :slight_smile: it’s working perfectly !! :rolleyes: Have a good day !!! :slight_smile:

Hey, Is there anywhere where I can download an example project? Thanks, Lewis

Update to 0.6.3
-Changed plugin to runtime to support shipping builds
-See the readme for shipping/packaging instructions.

This was happening because of launching v0.6.2 or lower version of the plugin which didn’t have packaged/shipping support. Download 0.6.3 plugin or later and see the readme for shipping/packaging instructions.

I don’t plan on including an example project as it is two simple steps to include leap support. Follow the how to instructions and you will be up and running in a few minutes.

Hi,

I tried running the plugin in UE4.5 and I get message:
The following modules are missing or build with a different engine version:
UE4Editor-LeapMotion.dll
Would you like to rebuild them now?

When I click yes, it fails to compile.
I copied Binaries and Plugin folders to the root of the project.
I also tried the 4.4 version fo the plugin and get a diffrent message
"Plugin ‘LeapMotion’ failed to load because module LeapMotion’ could be loaded…’

Any help will be much appreciated, I’m completly stuck! :frowning:

Ensure you’re on Win64, make sure you download the latest plugin version (0.6.3) this is compiled for UE4.5. When you play hit play not launch. If you want to run it outside the editor use Play->Standalone Game. If you want to package/cook and test for shipping, the shipping instructions.

Thanks for the quick reply. I’m on Win64. Downloaded the latest plugin and UE4.5. Unfortunately I can’t get to the editor (it just stops loading when the “The following modules are missing or build with a different engine version: UE4Editor-LeapMotion.dll Would you like to rebuild them now?” so I did not get to test it with play. Am I missing something?

Hey I’m putting the BP together but I noticed that I dont have a debug sphere at actor I can only bring up Draw Debug Sphere. Does that matter? Also I can only bring up stabilizedPalmPosition and stabilizedTipPosition, not palmPosition or tipPosition, I’m assuming you updated those nodes. Just thought I should mention.

UPDATE: Nevermind debug sphere at actor is a custom function that I didn’t scroll down far enough thank you though. I’m a newb so could you push me in the direction of adding actual bones to this system? Thanks ! great job!!!

Update to 0.6.4
-Fixed lack of categorization caused by updates in some of the blueprints classes
-Cleaned up a lot of the blueprint functions, turning all the relevant
ones into properties. This will reduce the number of execution pins.
-Things which return a new object generally remain a function for
potential performance reasons

Yeah something I did in the earlier updates changed the blueprint categorization. I went through each blueprint class in the plugin and changed a lot of them into properties which should reduce the execution pin requirements.

With the change I updated the quick setup and debug hand images which include each finger bone giving you something like this:


Grab the updated plugin (0.6.4) and check at the begining of the thread for the updated API example.

That error simply means your project is not in UE4.5, make sure to switch the engine version or create a new 4.5 project. I updated the plugin again, this time it was tested on a fresh no-code project and I confirmed it worked. Try re-downloading the latest plugin and start from the beginning.