[Plugin] Leap Motion - Event Driven

[spyro]

How to uninstall this plugin? I deleted all references, set the plugin to false in the config but still got a crash when restarting the project...

[getnamo]

Pardon my noob questions here, but I'm a bit confused on a few things.

What I'm trying to do is use my own skeleton and bind its bones to the leap motion hands. I'm looking at the way it was implemented for the BasicLeapCharacter blueprint through the LeapAnimBodyConnector.

The LeapAnimBodyConnector has this funky function called "ChangeBasis". It takes what looks like a pre-rotated value and does something with a post rotated value. I conceptually don't understand what's going on here and how it relates to the skel mesh of the character and the final output. For the left hand, the pre-rotation value is (-90,0,0) and the post rotation value is (0,0,90). How are you coming up with these values? What bones are you looking at?


I'm also really new to animation and I've never seen the FABRIK and Two Bone IK stuff. I'm guessing that the ChangeBasis method is related to the inverse kinematics stuff, but I'm a bit lost. The FABRIK documentation isn't very helpful in explaining what it does conceptually.

I'm glad you've asked the question, I was hoping someone would dig deep enough to ask about the implementation. The change basis function rotates all the AnimBody bones (that are parented, so if you rotate the body it will rotate all the bones) to a new basis. Pre-rotation applies input rotation before you're in the new space so it will affect all of your input rotations. In contrast post-rotation only rotates it by a constant amount and will not affect the basis of the input.

The reason for doing all of this is that in component space (e.g. character blueprints) X is forward Y is right, Z is up; but in animation Y is forward, X is left, Z is up. So in order for everything to appear correct we need to change the rotation space to the new Y = Forward orientation. That is all that function does.

Two-bone IK guesses a bone posture from a final position of the 'tip' bone. Typically you have another point where you encourage e.g. your elbow to bend towards. FABRIK on the other hand is a more general solver which can be have many more bones, so you can specify how long the bone chain is that gets affected by the 'tip' bone. In the leap rigging I currently have, I do not use Two-bone IK, instead we have only FABRIK nodes for spine/neck guessing for head positioning (from HMD) and shoulder guessing for your elbows. All other bones use the orientation values obtained from the leap motion. If we instead used IK for your hands, they would not always line up with what your real arms are doing because you would be disregarding the elbow and arm orientations that the leap motion sees for guessed positions. There are some minor problems with this approach, which boil down to mismatching arm and bone lengths, but for that you will always have some sort of compromise.

If you're trying to use your own skeleton, it may be worthwhile to look at animation re-targeting. Down the line I have to come up with a cleaner solution for replacing skeletons/meshes.

Getnamo,
AWESOME job on the physics addition....solved what I wanted to do and was helpful in learning more about the engine. Have you tried the leap image hands for unity? That would be amazing to get into this plug-in!! The actual image of your hands instead of the twitchy fingers is a presence enhancer for sure.

I am curious how hard you think this would be to implement.

I will be testing these assets soon and their implementation are in the pipeline

[Spittygood]

Getnamo,
AWESOME job on the physics addition....solved what I wanted to do and was helpful in learning more about the engine. Have you tried the leap image hands for unity? That would be amazing to get into this plug-in!! The actual image of your hands instead of the twitchy fingers is a presence enhancer for sure.

I am curious how hard you think this would be to implement.

[Slayemin]

Pardon my noob questions here, but I'm a bit confused on a few things.

What I'm trying to do is use my own skeleton and bind its bones to the leap motion hands. I'm looking at the way it was implemented for the BasicLeapCharacter blueprint through the LeapAnimBodyConnector.

The LeapAnimBodyConnector has this funky function called "ChangeBasis". It takes what looks like a pre-rotated value and does something with a post rotated value. I conceptually don't understand what's going on here and how it relates to the skel mesh of the character and the final output. For the left hand, the pre-rotation value is (-90,0,0) and the post rotation value is (0,0,90). How are you coming up with these values? What bones are you looking at?


I'm also really new to animation and I've never seen the FABRIK and Two Bone IK stuff. I'm guessing that the ChangeBasis method is related to the inverse kinematics stuff, but I'm a bit lost. The FABRIK documentation isn't very helpful in explaining what it does conceptually.

[astonish]

I'm an idiot, I somehow turned image pass through off in the control panel while re-configuring my LEAP. Oh well, great way to learn how everything is working inside by stepping through the blueprints. Good work! Thanks for the plug-in!

[astonish]

Is there a blueprint call to force the passthrough mode on when using the passthrough character? I want to set it to a debug key on my keyboard

Edit: I see there is a default key of "F" assigned to just such an operation, but it isn't working for me. Do I need to put the post process material in my scene? Something isn't working for me.

[commedieu]

Thanks so much for this Getnamo.

[getnamo]

Your plugin seems to work great but the leap itself is jittering like crazy. My hand is just jumping around and even simple gestures aren't recognized reliable. I tried to setup the debug circles, but left and right seems to be switched. Tried to mount the sensor the other way around, but didn't make any difference.

EDIT: Found the reason, the sensor was not configured in HMD mode:

This fixed it:

...

Yep the 'Use Topdown' tickbox does all the transformations necessary for the leap to be used in VR and all your reported positions should be 1:1 with the real world.

If the leap is jittering, you may want to ensure you don't have extra infrared light sources such as sunlight, additionally the leap will perform poorly if you have tables and objects very close to the sensor.

Has anyone tried the Leap Motion UE4 plugin with Android the GearVR? (where the new GearVR has a USB port). It sounds like there is an Android SDK for Leap Motion: https://community.leapmotion.com/t/i...axy-note4/2753 but is it included already in the UE4 plugin??

This plugin doesn't support it out of the box, but you have full source to attempt a bind yourself. Make sure to make a pull request if you get it working!



New Learning resources, full example project!

Check out the latest leap motion blog entry for a run down on a full example of how to use the plugin to make cool things in VR!

The three tutorial videos contain parts on how to implement basic jenga, jenga with pickup/drop cabilitilies, and finally telekinesis powers.

Part 1 - Setup and Collision

What you'll learn from the first part:

Drag and drop installation


Selecting a leap character and hitting Play to see interactive hands.



Build Jenga stacks and push them around with the Leap Collision Character.



Full instructions available in the video below

Part 2 - Picking up and dropping objects
Dig into the second part and learn to extend functionality with input mapping in order to pickup and drop blocks

Part 3 - Raytracing and Telekinesis
Learn how to do raytracing in UE4



Enable your telekinetic powers to pickup blocks from the ground and place them back on top of the stack!

Resources
When you've finished you can compare your results to the final project zip, or use it as a reference!

If you just want to try the demo, grab the executable instead.

If you want to see an example of leap-only locomotion try the mage arena game.


Let me know if you've found any of this helpful

[spyro]

Your plugin seems to work great but the leap itself is jittering like crazy. My hand is just jumping around and even simple gestures aren't recognized reliable. I tried to setup the debug circles, but left and right seems to be switched. Tried to mount the sensor the other way around, but didn't make any difference.

EDIT: Found the reason, the sensor was not configured in HMD mode:

This fixed it:

[aussieburger]

Has anyone tried the Leap Motion UE4 plugin with Android the GearVR? (where the new GearVR has a USB port). It sounds like there is an Android SDK for Leap Motion: https://community.leapmotion.com/t/i...axy-note4/2753 but is it included already in the UE4 plugin??

[getnamo]

I'm using the LeapRiggedCharacter but want to put the Leap stationary in front of me. I know OptimizeForHMD should be used for this. Can someone help me with:
1. Which settings to use for OptimizeForHMD when using the Leap placed on a table.
2. Where in the LeapRiggedCharacter is OptimizeForHMD called, so I know where to change the settings.

Thanks in advance!

If you're using Leap Rigged Character, then it automatically supports both VR mode and regular mode. If you wish to adjust the Leap base offset (where the controller is on your table), modify the position of the DefaultModeSprite in the LeapAnimBodyConnector, imagine that sprite as your virtual leap motion controller.


All leap-specific parts for convenience characters can be found under LeapAnimBodyConnector.

Hi getnamo, did you manage to reproduce this issue on a mac?

I sadly don't have the resources to support the mac version more often than once per engine release. This will be inspected when I compile the plugin for 4.8.

I just wanted to jump in and say that this looks fantastic. Great job, getnamo!

Thanks! Glad you like it

As little help for those who have had a few issues getting the leap motion to work with UE4.7 here's a few tips that I found helpful. Firstly once you have copied the files into the root directory, when you load UE4 check the 0.9... version of the leap rather than v1.0.

...

Then once inside UE4, delete the character start model so that you can see you hands inside the environment.

... to ...

Once these extra steps are taken, the leap motion should work inside you environment.

This is very helpful, I will point future questions of the sort to your post.

I'm super new to this, though I kind of wish I weren't. How can I create a reference to the leap character? For instance, If I want to parent an object to the rigged hands, I'd need to create a reference to the object, yes?

But for some reason, I can't seem to figure this out.

The convenience assets are not that easy to decode, because they're meant to separate the input hardware from the actual visual/physics assets. That said if you want to attach something to the hand, simply find the bone or socket you want an item to be attached to and attach it to the mesh. You don't have to worry about the leap motion at all and simply interact with your character mesh as if it is another UE pawn/character knowing it will track your hands when they're visible.

If you want something leap specific such as their supported gesture events, you can follow the github documentation on adding a leap component to any blueprint you wish to respond to the leap events (see heading How to use it - Blueprint without Convenience Content, Quick Setup). You can mix and match convenience with custom as the plugin supports multi-casting leap components. The github repository is your how-to for everything related to this unofficial leap motion plugin

[IKnowBashFu]

I'm super new to this, though I kind of wish I weren't. How can I create a reference to the leap character? For instance, If I want to parent an object to the rigged hands, I'd need to create a reference to the object, yes?

But for some reason, I can't seem to figure this out.

[BunHead]

As little help for those who have had a few issues getting the leap motion to work with UE4.7 here's a few tips that I found helpful. Firstly once you have copied the files into the root directory, when you load UE4 check the 0.9... version of the leap rather than v1.0.



Then once inside UE4, delete the character start model so that you can see you hands inside the environment.

to

Once these extra steps are taken, the leap motion should work inside you environment.

[gmpreussner]

I just wanted to jump in and say that this looks fantastic. Great job, getnamo!

[HavocX]

I'm using the LeapRiggedCharacter but want to put the Leap stationary in front of me. I know OptimizeForHMD should be used for this. Can someone help me with:
1. Which settings to use for OptimizeForHMD when using the Leap placed on a table.
2. Where in the LeapRiggedCharacter is OptimizeForHMD called, so I know where to change the settings.

Thanks in advance!