[Plugin] Leap Motion - Event Driven

[getnamo]

@Joe1029;166913

Has anyone been working on setting leap hands to mesh? I went to the hydra thread like said and studied the examples him and some other users posted along with the IK and animation docs. I thought I had an idea how to transport what I had seen in that thread over to the leap but I failed quite hilariously instead of following my hands like I hoped it would. Once either of my hands are detected myCharacter’s hands shoot straight to its crotch pics below. Since I’m an all around noob for the moment I couldnt place the pictures like I wanted to but I will explain my process.

First pic shows the event graph in my BP_LeapController. In it I made two events; Left Hand moved and Right hand Moved, to be honest I don’t know why I set it up this way cuz all it is doing is drawing my spheres (I think) and then I Set Leap enabled. The Draw Debug Sphere function is exactly the same as example.

Second picture is the HeroTPP_AnimBP’s event graph, here I was hoping I would be able to get the location from BP_leapController such as left Wrist and right Wrist and SET the LeftHand and RightHand location. I think I’m not receiving any data at this point which would make sense why myCharacter is touching itself. (The wires outside the pic are from the regular animBP.)

After that I set up both IK nodes for the left and right hand but since I don’t really know if I’m receiving any data from the initial cast I am stumped if anything in here is messed up or not. Do I need to set up rotation?

If anybody has a clue or is interested in figuring this out I would be happy to work together I just want to be able to see my hands in VR

Your graph looks ok except for two things.

One your Joint Target is 0,0,0 this is probably incorrect, joint target should be in the direction of your elbow, you can play around with it in your Persona Preview to find the right values.

The second part is that you’re forwarding the raw Leap Values. Consider for a moment what these are: the position of the hand in relation to the leap motion controller in the real world. What the animation graph is looking for is the position of the hands in relation to the character root component position. So you need to find the offset between Leap Real World->Character Root. If you’re using the Leap default mode this offset is unknown. With the hydra we solved this using a calibration step which gave the offset of the hydra base to the shoulder midpoint which then got shifted down to the character root. This isn’t really possible with the Leap in default mode since there isn’t a calibration step which can determine where you body is in relation to the leap (maybe you could point toward your body and grab that point, but that sounds shaky at best), so your best bet is to have a fixed offset that the user can adjust.

However: If you’re using the Leap Motion mounted to your HMD, this is moot. Simply tick optimize for hmd, auto-shift, and auto-rotate and all your values will be in 1:1 in component space. This means you can forward the raw values straight to the skeleton. The reason this works is because it is mounted to your HMD which is tracked with a direct position, so the plugin figures out the offset and rotation adjustment to translate your values into component space automagically :rolleyes:.

@bullale;166803

Thank you. I was able to compile it with Unreal Engine 4.5.1

I noticed that the event executions aren’t visualized in the blueprint view. I don’t know if that’s expected or not.

Do you know how I can find the real leap-character offset for use with the oculus rift? I can’t seem to find a setting that feels natural. I still haven’t figured out the character/actor/camera hierarchy.
Do you have a DK2? If you do and it works well for you, would you be able to share your character BluePrint in your repo? I’ll keep working at it but any help is greatly appreciated.

To my knowledge event visualization (blueprint debugging) doesn’t happen for interfaces, ask epic to add that capability.

Currently I do not have a character blueprint specific to the leap, I’m working on something more generalized which would allow a lot of hardware specific binding to be skipped, this may take some time though, until then explore the setup with the rest of the ue4 leapers

NB: General Hierarchy for character animation is Character/Pawn: Leap Controller Event/Poll->OffsetValues, Animation Event Graph: Character:OffsetValues->Hand/Finger Values in Component Space, Animation Graph: Hand/Finger Values->IK Bone Effector location/FK bone rotation

@joemamaishere123;166936

…

Hey guys, long time listener, first time caller.
I was able to install the plug-in just fine.
In the project, I was able to create via blueprint the Debug Hands example from the page.
However, once I exit the project and try to re-open it, the editor crashes, and I get the following error message.
Any help would be super appreciated. Have a nice day!

From your crash output it doesn’t seem to be related to the plugin. If you still suspect the plugin, you can compile the plugin (follow instructions earlier in the thread) to get a .pdb file which will give you more detailed output in your logs and in the crash output.