[Plugin] Leap Motion - Event Driven

[epetije]

Hi,

first of all thanx for your work.
I'm trying to use your plugin and so far everything's fine but...
How can I intercept a SwipeGesture and use its variables in my blueprint ? I looked quickly in your C++ and found nothing.
Is there a way ?

[getnamo]

Well I am practically 100% certain the error is coming from the plugin because everything works fine until I add LeapMotion blueprints into the event graph. It can be anything as simple as a single LeapEvent, and the editor crashes with the above message.

However, I may have found out why I am getting the above crazy error message.
I tried to add the LeapEventInterface without adding any blueprint code, and I get the following error when re-launching the editor.

[ATTACH=CONFIG]14765[/ATTACH]

The LeapEventInterface is then no longer included in the implemented interfaces.
So I hypothesize that... the LeapEventInterface keeps getting removed and then when UE4 tries to "read" the LeapMotion blueprints, it crashes because the interface isn't there.
Do you have any idea why I am getting this new error message? And why the LeapEventInterface is being removed?

I think you might have some left over functions hanging around in the event graph. When you remove LeapEventInterface from a blueprint that depends on it, it gives you an option to remove them or convert to custom nodes. Then when you add the interface again it will clash since the names are the same.

I recommend starting from a fresh blueprint (copy a character) and add/remove things until you stop getting these errors. I've tried but I cannot reproduce it myself, try checking your logs found in {Project Root}/saved/logs. By default the plugin warns you if you have a Leap Controller component, but no interface and having just an interface is harmless as it would simply not cause your events to be called.

[joemamaishere123]

Well I am practically 100% certain the error is coming from the plugin because everything works fine until I add LeapMotion blueprints into the event graph. It can be anything as simple as a single LeapEvent, and the editor crashes with the above message.

However, I may have found out why I am getting the above crazy error message.
I tried to add the LeapEventInterface without adding any blueprint code, and I get the following error when re-launching the editor.



The LeapEventInterface is then no longer included in the implemented interfaces.
So I hypothesize that... the LeapEventInterface keeps getting removed and then when UE4 tries to "read" the LeapMotion blueprints, it crashes because the interface isn't there.
Do you have any idea why I am getting this new error message? And why the LeapEventInterface is being removed?

[getnamo]

Has anyone been working on setting leap hands to mesh? I went to the hydra thread like getnamo said and studied the examples him and some other users posted along with the IK and animation docs. I thought I had an idea how to transport what I had seen in that thread over to the leap but I failed quite hilariously instead of following my hands like I hoped it would. Once either of my hands are detected myCharacter's hands shoot straight to its crotch pics below. Since I'm an all around noob for the moment I couldnt place the pictures like I wanted to but I will explain my process.

First pic shows the event graph in my BP_LeapController. In it I made two events; Left Hand moved and Right hand Moved, to be honest I don't know why I set it up this way cuz all it is doing is drawing my spheres (I think) and then I Set Leap enabled. The Draw Debug Sphere function is exactly the same as @Getnamo example.

Second picture is the HeroTPP_AnimBP's event graph, here I was hoping I would be able to get the location from BP_leapController such as left Wrist and right Wrist and SET the LeftHand and RightHand location. I think I'm not receiving any data at this point which would make sense why myCharacter is touching itself. (The wires outside the pic are from the regular animBP.)

After that I set up both IK nodes for the left and right hand but since I don't really know if I'm receiving any data from the initial cast I am stumped if anything in here is messed up or not. Do I need to set up rotation?

If anybody has a clue or is interested in figuring this out I would be happy to work together I just want to be able to see my hands in VR

Your graph looks ok except for two things.

One your Joint Target is 0,0,0 this is probably incorrect, joint target should be in the direction of your elbow, you can play around with it in your Persona Preview to find the right values.

The second part is that you're forwarding the raw Leap Values. Consider for a moment what these are: the position of the hand in relation to the leap motion controller in the real world. What the animation graph is looking for is the position of the hands in relation to the character root component position. So you need to find the offset between Leap Real World->Character Root. If you're using the Leap default mode this offset is unknown. With the hydra we solved this using a calibration step which gave the offset of the hydra base to the shoulder midpoint which then got shifted down to the character root. This isn't really possible with the Leap in default mode since there isn't a calibration step which can determine where you body is in relation to the leap (maybe you could point toward your body and grab that point, but that sounds shaky at best), so your best bet is to have a fixed offset that the user can adjust.

However: If you're using the Leap Motion mounted to your HMD, this is moot. Simply tick optimize for hmd, auto-shift, and auto-rotate and all your values will be in 1:1 in component space. This means you can forward the raw values straight to the skeleton. The reason this works is because it is mounted to your HMD which is tracked with a direct position, so the plugin figures out the offset and rotation adjustment to translate your values into component space automagically .

Thank you. I was able to compile it with Unreal Engine 4.5.1

I noticed that the event executions aren't visualized in the blueprint view. I don't know if that's expected or not.

Do you know how I can find the real leap-character offset for use with the oculus rift? I can't seem to find a setting that feels natural. I still haven't figured out the character/actor/camera hierarchy.
Do you have a DK2? If you do and it works well for you, would you be able to share your character BluePrint in your github repo? I'll keep working at it but any help is greatly appreciated.

To my knowledge event visualization (blueprint debugging) doesn't happen for interfaces, ask epic to add that capability.

Currently I do not have a character blueprint specific to the leap, I'm working on something more generalized which would allow a lot of hardware specific binding to be skipped, this may take some time though, until then explore the setup with the rest of the ue4 leapers

NB: General Hierarchy for character animation is Character/Pawn: Leap Controller Event/Poll->OffsetValues, Animation Event Graph: Character:OffsetValues->Hand/Finger Values in Component Space, Animation Graph: Hand/Finger Values->IK Bone Effector location/FK bone rotation

...

Hey guys, long time listener, first time caller.
I was able to install the plug-in just fine.
In the project, I was able to create via blueprint the Debug Hands example from the github page.
However, once I exit the project and try to re-open it, the editor crashes, and I get the following error message.
Any help would be super appreciated. Have a nice day!

From your crash output it doesn't seem to be related to the plugin. If you still suspect the plugin, you can compile the plugin (follow instructions earlier in the thread) to get a .pdb file which will give you more detailed output in your logs and in the crash output.

[joemamaishere123]

Hey guys, long time listener, first time caller.
I was able to install the plug-in just fine.
In the project, I was able to create via blueprint the Debug Hands example from the github page.
However, once I exit the project and try to re-open it, the editor crashes, and I get the following error message.
Any help would be super appreciated. Have a nice day!

[Joe1029]

Has anyone been working on setting leap hands to mesh? I went to the hydra thread like getnamo said and studied the examples him and some other users posted along with the IK and animation docs. I thought I had an idea how to transport what I had seen in that thread over to the leap but I failed quite hilariously instead of following my hands like I hoped it would. Once either of my hands are detected myCharacter's hands shoot straight to its crotch pics below. Since I'm an all around noob for the moment I couldnt place the pictures like I wanted to but I will explain my process.

First pic shows the event graph in my BP_LeapController. In it I made two events; Left Hand moved and Right hand Moved, to be honest I don't know why I set it up this way cuz all it is doing is drawing my spheres (I think) and then I Set Leap enabled. The Draw Debug Sphere function is exactly the same as @Getnamo example.

Second picture is the HeroTPP_AnimBP's event graph, here I was hoping I would be able to get the location from BP_leapController such as left Wrist and right Wrist and SET the LeftHand and RightHand location. I think I'm not receiving any data at this point which would make sense why myCharacter is touching itself. (The wires outside the pic are from the regular animBP.)

After that I set up both IK nodes for the left and right hand but since I don't really know if I'm receiving any data from the initial cast I am stumped if anything in here is messed up or not. Do I need to set up rotation?

If anybody has a clue or is interested in figuring this out I would be happy to work together I just want to be able to see my hands in VR

[rasamaya]

Nice work on the video input, I will be testing tonight, just got my leap today!

[Joe1029]

Bullale I got the arms feeling natural in the oculus by playing with the vector values inside of the "draw sphere at actor" I think those values were around x=40 y=0= z= 40, the values in the examples were throwing my hands way down.

[bullale]

To build the plugin from source, simply install the plugin as usual (drag and drop Plugins and Binaries into project root) and add code to your project. Delete the UE4Editor-LeapMotion.dll and then when you recompile it should include your plugin in the compilation as well.

Thank you. I was able to compile it with Unreal Engine 4.5.1

I noticed that the event executions aren't visualized in the blueprint view. I don't know if that's expected or not.

Do you know how I can find the real leap-character offset for use with the oculus rift? I can't seem to find a setting that feels natural. I still haven't figured out the character/actor/camera hierarchy.
Do you have a DK2? If you do and it works well for you, would you be able to share your character BluePrint in your github repo? I'll keep working at it but any help is greatly appreciated.

[getnamo]

I have the same problem as BartBarlowski.

I'm on Win 8.1 x64. My Editor is built from source. It says it is version 4.5.1. Anytime I add your folders to a project I can no longer open that project because of the above error.
I'm now downloading the Engine provided by the launcher (4.5.0). I'll let you know if that makes a difference.

However, the better solution would be if I could build the plugin myself. Can you please provide instructions on how to do that?

Thank you!

To build the plugin from source, simply install the plugin as usual (drag and drop Plugins and Binaries into project root) and add code to your project. Delete the UE4Editor-LeapMotion.dll and then when you recompile it should include your plugin in the compilation as well.

Alternatively you can add the plugin into the engine root in the same way (turning it into an engine plugin) and it will be enabled in all projects by default, but I haven't tested this myself.

[bullale]

Hi,

I tried running the plugin in UE4.5 and I get message:
The following modules are missing or build with a different engine version:
UE4Editor-LeapMotion.dll
Would you like to rebuild them now?

I have the same problem as BartBarlowski.

I'm on Win 8.1 x64. My Editor is built from source. It says it is version 4.5.1. Anytime I add your folders to a project I can no longer open that project because of the above error.
I'm now downloading the Engine provided by the launcher (4.5.0). I'll let you know if that makes a difference.

However, the better solution would be if I could build the plugin myself. Can you please provide instructions on how to do that?

Thank you!

[getnamo]

Hi Getnamo.

Great work on the plugin. Is there an example blueprint for using the skeletal mesh of the hero character instead of the debug spheres?

Thanks

See the hydra thread where we've talked about inverse kinematics to set this up for hands/arms. It is not a trivial topic, so ensure you understand how the unreal engine animation system works, essentially you just forward your leap motion positions to IK nodes (or FK rotations) in the animation graph, refer to IK setup documentation and animation documentation in general for details.

[Synesthesia]

Hi Getnamo.

Great work on the plugin. Is there an example blueprint for using the skeletal mesh of the hero character instead of the debug spheres?

Thanks

[getnamo]

Updated the image for recognizing gestures which reflects the new property based setup. With the latest plugin version, just follow this graph:

[Joe1029]

Has anyone been able to get the gesture sample hooked up? I can't find where "TYPE" is. I would really appreciate any help.