[Plugin] Leap Motion - Event Driven

[bullale]

To build the plugin from source, simply install the plugin as usual (drag and drop Plugins and Binaries into project root) and add code to your project. Delete the UE4Editor-LeapMotion.dll and then when you recompile it should include your plugin in the compilation as well.

Thank you. I was able to compile it with Unreal Engine 4.5.1

I noticed that the event executions aren't visualized in the blueprint view. I don't know if that's expected or not.

Do you know how I can find the real leap-character offset for use with the oculus rift? I can't seem to find a setting that feels natural. I still haven't figured out the character/actor/camera hierarchy.
Do you have a DK2? If you do and it works well for you, would you be able to share your character BluePrint in your github repo? I'll keep working at it but any help is greatly appreciated.

[Joe1029]

Bullale I got the arms feeling natural in the oculus by playing with the vector values inside of the "draw sphere at actor" I think those values were around x=40 y=0= z= 40, the values in the examples were throwing my hands way down.

[rasamaya]

Nice work on the video input, I will be testing tonight, just got my leap today!

[Joe1029]

Has anyone been working on setting leap hands to mesh? I went to the hydra thread like getnamo said and studied the examples him and some other users posted along with the IK and animation docs. I thought I had an idea how to transport what I had seen in that thread over to the leap but I failed quite hilariously instead of following my hands like I hoped it would. Once either of my hands are detected myCharacter's hands shoot straight to its crotch pics below. Since I'm an all around noob for the moment I couldnt place the pictures like I wanted to but I will explain my process.

First pic shows the event graph in my BP_LeapController. In it I made two events; Left Hand moved and Right hand Moved, to be honest I don't know why I set it up this way cuz all it is doing is drawing my spheres (I think) and then I Set Leap enabled. The Draw Debug Sphere function is exactly the same as @Getnamo example.

Second picture is the HeroTPP_AnimBP's event graph, here I was hoping I would be able to get the location from BP_leapController such as left Wrist and right Wrist and SET the LeftHand and RightHand location. I think I'm not receiving any data at this point which would make sense why myCharacter is touching itself. (The wires outside the pic are from the regular animBP.)

After that I set up both IK nodes for the left and right hand but since I don't really know if I'm receiving any data from the initial cast I am stumped if anything in here is messed up or not. Do I need to set up rotation?

If anybody has a clue or is interested in figuring this out I would be happy to work together I just want to be able to see my hands in VR

[joemamaishere123]

Hey guys, long time listener, first time caller.
I was able to install the plug-in just fine.
In the project, I was able to create via blueprint the Debug Hands example from the github page.
However, once I exit the project and try to re-open it, the editor crashes, and I get the following error message.
Any help would be super appreciated. Have a nice day!

[getnamo]

Has anyone been working on setting leap hands to mesh? I went to the hydra thread like getnamo said and studied the examples him and some other users posted along with the IK and animation docs. I thought I had an idea how to transport what I had seen in that thread over to the leap but I failed quite hilariously instead of following my hands like I hoped it would. Once either of my hands are detected myCharacter's hands shoot straight to its crotch pics below. Since I'm an all around noob for the moment I couldnt place the pictures like I wanted to but I will explain my process.

First pic shows the event graph in my BP_LeapController. In it I made two events; Left Hand moved and Right hand Moved, to be honest I don't know why I set it up this way cuz all it is doing is drawing my spheres (I think) and then I Set Leap enabled. The Draw Debug Sphere function is exactly the same as @Getnamo example.

Second picture is the HeroTPP_AnimBP's event graph, here I was hoping I would be able to get the location from BP_leapController such as left Wrist and right Wrist and SET the LeftHand and RightHand location. I think I'm not receiving any data at this point which would make sense why myCharacter is touching itself. (The wires outside the pic are from the regular animBP.)

After that I set up both IK nodes for the left and right hand but since I don't really know if I'm receiving any data from the initial cast I am stumped if anything in here is messed up or not. Do I need to set up rotation?

If anybody has a clue or is interested in figuring this out I would be happy to work together I just want to be able to see my hands in VR

Your graph looks ok except for two things.

One your Joint Target is 0,0,0 this is probably incorrect, joint target should be in the direction of your elbow, you can play around with it in your Persona Preview to find the right values.

The second part is that you're forwarding the raw Leap Values. Consider for a moment what these are: the position of the hand in relation to the leap motion controller in the real world. What the animation graph is looking for is the position of the hands in relation to the character root component position. So you need to find the offset between Leap Real World->Character Root. If you're using the Leap default mode this offset is unknown. With the hydra we solved this using a calibration step which gave the offset of the hydra base to the shoulder midpoint which then got shifted down to the character root. This isn't really possible with the Leap in default mode since there isn't a calibration step which can determine where you body is in relation to the leap (maybe you could point toward your body and grab that point, but that sounds shaky at best), so your best bet is to have a fixed offset that the user can adjust.

However: If you're using the Leap Motion mounted to your HMD, this is moot. Simply tick optimize for hmd, auto-shift, and auto-rotate and all your values will be in 1:1 in component space. This means you can forward the raw values straight to the skeleton. The reason this works is because it is mounted to your HMD which is tracked with a direct position, so the plugin figures out the offset and rotation adjustment to translate your values into component space automagically .

Thank you. I was able to compile it with Unreal Engine 4.5.1

I noticed that the event executions aren't visualized in the blueprint view. I don't know if that's expected or not.

Do you know how I can find the real leap-character offset for use with the oculus rift? I can't seem to find a setting that feels natural. I still haven't figured out the character/actor/camera hierarchy.
Do you have a DK2? If you do and it works well for you, would you be able to share your character BluePrint in your github repo? I'll keep working at it but any help is greatly appreciated.

To my knowledge event visualization (blueprint debugging) doesn't happen for interfaces, ask epic to add that capability.

Currently I do not have a character blueprint specific to the leap, I'm working on something more generalized which would allow a lot of hardware specific binding to be skipped, this may take some time though, until then explore the setup with the rest of the ue4 leapers

NB: General Hierarchy for character animation is Character/Pawn: Leap Controller Event/Poll->OffsetValues, Animation Event Graph: Character:OffsetValues->Hand/Finger Values in Component Space, Animation Graph: Hand/Finger Values->IK Bone Effector location/FK bone rotation

...

Hey guys, long time listener, first time caller.
I was able to install the plug-in just fine.
In the project, I was able to create via blueprint the Debug Hands example from the github page.
However, once I exit the project and try to re-open it, the editor crashes, and I get the following error message.
Any help would be super appreciated. Have a nice day!

From your crash output it doesn't seem to be related to the plugin. If you still suspect the plugin, you can compile the plugin (follow instructions earlier in the thread) to get a .pdb file which will give you more detailed output in your logs and in the crash output.

[joemamaishere123]

Well I am practically 100% certain the error is coming from the plugin because everything works fine until I add LeapMotion blueprints into the event graph. It can be anything as simple as a single LeapEvent, and the editor crashes with the above message.

However, I may have found out why I am getting the above crazy error message.
I tried to add the LeapEventInterface without adding any blueprint code, and I get the following error when re-launching the editor.



The LeapEventInterface is then no longer included in the implemented interfaces.
So I hypothesize that... the LeapEventInterface keeps getting removed and then when UE4 tries to "read" the LeapMotion blueprints, it crashes because the interface isn't there.
Do you have any idea why I am getting this new error message? And why the LeapEventInterface is being removed?

[getnamo]

Well I am practically 100% certain the error is coming from the plugin because everything works fine until I add LeapMotion blueprints into the event graph. It can be anything as simple as a single LeapEvent, and the editor crashes with the above message.

However, I may have found out why I am getting the above crazy error message.
I tried to add the LeapEventInterface without adding any blueprint code, and I get the following error when re-launching the editor.

[ATTACH=CONFIG]14765[/ATTACH]

The LeapEventInterface is then no longer included in the implemented interfaces.
So I hypothesize that... the LeapEventInterface keeps getting removed and then when UE4 tries to "read" the LeapMotion blueprints, it crashes because the interface isn't there.
Do you have any idea why I am getting this new error message? And why the LeapEventInterface is being removed?

I think you might have some left over functions hanging around in the event graph. When you remove LeapEventInterface from a blueprint that depends on it, it gives you an option to remove them or convert to custom nodes. Then when you add the interface again it will clash since the names are the same.

I recommend starting from a fresh blueprint (copy a character) and add/remove things until you stop getting these errors. I've tried but I cannot reproduce it myself, try checking your logs found in {Project Root}/saved/logs. By default the plugin warns you if you have a Leap Controller component, but no interface and having just an interface is harmless as it would simply not cause your events to be called.

[epetije]

Hi,

first of all thanx for your work.
I'm trying to use your plugin and so far everything's fine but...
How can I intercept a SwipeGesture and use its variables in my blueprint ? I looked quickly in your C++ and found nothing.
Is there a way ?

[joemamaishere123]

Oh no, there are no leftover functions in the event graph or anything. It's a fresh new project. It's a new FPS C++ template project. I go into the MyCharacter blueprint and ONLY try to add the LeapEventInterface in implemented interfaces. It saves and compiles fine. But then when I exit the project, and re-open it, I get the above error.

Later, if I try to add even a single LeapEvent or anything to do with the LeapMotion in the EventGraph, it, too, saves and compiles fine. But upon exiting and trying to re-open I get the original error where the editor doesn't even open.

I realize if you can't reproduce the error, there is not much you can do. But hopefully you can help because this is maddening.

I should note the LeapEventInterface isn't being removed by ME. I receive the above error message and the LeapInterface is automatically removed.

[zalo]

Gah, this plug-in so so cool, but I can't get my favorite part to work!

Whenever I try to access an image, this appears for every frame in the output:

[2014.10.21-18.20.34:842][101]LogClass:Error: Warning! Leap Image SDK access is denied, please enable image support from the Leap Controller.

I have image support enabled of course; all of my Unity apps work. Disabling Image support has no effect.

All non-image functions seem to work fine (fingertip positions, etc), but I was really hoping to make a kick-*** passthrough demo...

EDIT: Hmm, it seems I'm using a slightly newer version (2.1.5+22699)... drat they don't have the old one online.

Maybe updating the .dll will help...

[getnamo]

Oh no, there are no leftover functions in the event graph or anything. It's a fresh new project. It's a new FPS C++ template project. I go into the MyCharacter blueprint and ONLY try to add the LeapEventInterface in implemented interfaces. It saves and compiles fine. But then when I exit the project, and re-open it, I get the above error.

Later, if I try to add even a single LeapEvent or anything to do with the LeapMotion in the EventGraph, it, too, saves and compiles fine. But upon exiting and trying to re-open I get the original error where the editor doesn't even open.

I realize if you can't reproduce the error, there is not much you can do. But hopefully you can help because this is maddening.

I should note the LeapEventInterface isn't being removed by ME. I receive the above error message and the LeapInterface is automatically removed.

I was able to reproduce it following what you said, it seems to be related to C++ code as a base vs a blueprint one, will check around what causes this. This shouldn't be happening and it appears this type of problem is more widespread than just this plugin: https://answers.unrealengine.com/que...reeeditor.html.

Will let you know if I find a fix.

[getnamo]

Gah, this plug-in so so cool, but I can't get my favorite part to work!

Whenever I try to access an image, this appears for every frame in the output:


I have image support enabled of course; all of my Unity apps work. Disabling Image support has no effect.

All non-image functions seem to work fine (fingertip positions, etc), but I was really hoping to make a kick-*** passthrough demo...

You need to make sure you're on the 2.1.5 sdk and you enable it from the blueprint via LeapController->enable image support. Typically you'd call this at begin play or another init type function. This follows how the Leap API expects this to work, probably an optimization to not use the extra bandwidth when not needed.

[Joe1029]

Hey Getnamo thanks for the advice. So am I correct in using the wrist position for this? I am using an oculus and making this game for oculus so I do have the HMD optimized ticked. I checked out the values for my lowerhand_L but I don't know what you mean by forward direction of the joint target location. Could you explain a little more?

[getnamo]

Hey Getnamo thanks for the advice. So am I correct in using the wrist position for this? I am using an oculus and making this game for oculus so I do have the HMD optimized ticked. I checked out the values for my lowerhand_L but I don't know what you mean by forward direction of the joint target location. Could you explain a little more?

Depending on what you're trying to do, wrist can work, but so can the hand->palm position which is more appropriate for hand position.

In IK setups you need a joint target. Generally these are 2-bone IKs and they behave somewhat like an elbow or knee type joint does on a person. So for the IK setup to behave properly you need your joint target be 'behind' your elbow, so that when the positions cause the bones to bend it will bend towards the direction of the elbow. Play around with the values in persona to find the sweet spot for the joint target.

Oh no, there are no leftover functions in the event graph or anything. It's a fresh new project. It's a new FPS C++ template project. I go into the MyCharacter blueprint and ONLY try to add the LeapEventInterface in implemented interfaces. It saves and compiles fine. But then when I exit the project, and re-open it, I get the above error.

Later, if I try to add even a single LeapEvent or anything to do with the LeapMotion in the EventGraph, it, too, saves and compiles fine. But upon exiting and trying to re-open I get the original error where the editor doesn't even open.

I realize if you can't reproduce the error, there is not much you can do. But hopefully you can help because this is maddening.

I should note the LeapEventInterface isn't being removed by ME. I receive the above error message and the LeapInterface is automatically removed.

Ok found a temporary workaround. Create a fps template, duplicate the MyCharacter in case you mess up in the future (as far as I know the bp is not recoverable after you add the component or interface), now for the fix:

1.Sub-class the Mycharacter in blueprint which by default will make a MyCharacter_Child.
2.Add your leap interface and leap component to this child class.

This will put a layer of indirection to your C++ class and the leap blueprint and it ... works.

I believe this stems from the origins of the plugin which retained a direct link to the Leap header and put all the classes into private implementation. These are still visible in blueprint, but I think C++ cannot include them properly. I will need to rethink how the architecture is built and re-implement it in such a way that the classes can be publicly exposed and included in C++ source. Until then just sub-class your C++ class and use that child to implement leap functions.