[Plugin] Leap Motion - Event Driven

[zalo]

The reason the distortion maps appear in the green and blue channels has to do with the pixel format you're using ("PF_B8G8R8A8") in the CreateTransient function (orders it BGRA).

You want to use "PF_R8G8B8A8" to get them into proper RGBA... though you should really be using "PF_G32R32F" (32-Bit Floats in each of the Red and Green channels) for lossless distortion data.

EDIT: Hmm, looks like my bit-fu isn't up to the task...

[BartBarlowski]

Finally got it work! I had to manually copy Leap.dll from the Win64 folder to my Windows/System32 folder. You should add this to the install instructions.

[getnamo]

Ok Getnamo update on my failures. In an attempt to get the joint target set up I created sockets at each elbow and placed the sockets 60 units behind each elbow (respectively 60 and -60) this is what I understood as behind the elbow. Then I put those value in the Y axis' of my joint variables plugged that into the joint target location and still nothing. I notice that when I launch the game the characters hands are still around its waist but the hands are receiving input from the leap so they move around near the waist but do not lift up . My hands are not moving in the z axis.

Couple of things (and this is why skeletal animation isn't really a trivial topic). Make sure you have rotated your inputs to have Y as forward (whereas world uses X), make sure again you are not just forwarding the raw values, but that you have a leap to character root offset added to the inputs before they are forwarded (unless you're using topdown with an hmd). The joint targets should probably be in component space and should not rotate with your arm movements (it may work otherwise, but I haven't personally tested this)

In your preview it appears that your hands are close to your character pelvis which indicates the alpha is working, since 0,0,0 is your component root which in the default character is at the pelvis. Now all you need to do to preview is change the Left Hand and Right Hand Position and see how the elbow bends as you move the positions around. Adjust the joint targets such that the elbow bends in a natural way.

Also ideally you should set Alpha to 0 whenever you hands are not visible, since you will have wrong data otherwise. That way you can let your animations handle your hand positions when they aren't being tracked by the leap. Quite a lot of this is covered in the hydra thread. Debug your vectors to visualize what they are doing and try to make them match what the animation system is expecting (by seeing what preview values are needed for the hand motions), this will indicate what you're doing wrong.

I am working on a generalized structure, which will have a convenience skeletal bind which will work for all vr input types, but it may be a few weeks before that gets posted, until then refer to the past few posts, the hydra thread and the unreal IK documentation for the how-to knowledge.

[Joe1029]

Ok Getnamo update on my failures. In an attempt to get the joint target set up I created sockets at each elbow and placed the sockets 60 units behind each elbow (respectively 60 and -60) this is what I understood as behind the elbow. Then I put those value in the Y axis' of my joint variables plugged that into the joint target location and still nothing. I notice that when I launch the game the characters hands are still around its waist but the hands are receiving input from the leap so they move around near the waist but do not lift up . My hands are not moving in the z axis.

[Joe1029]

So I set up leftJoint and rightJoint variables and I was tweaking the numbers but nothing happens and I made sure my alpha was on which is being driven by Bool LeapEnabled which is checked. Is it possible that something else is broken?

[knack]

hey anyone can make a small video for show the actual and real latency?

[getnamo]

Ok when you say play around with the values in persona do you mean I go into persona find the location of my characters elbows (lowerhand_l and lowerhand_r) and plug those in as join target location?

I get movement out of my hands but theyre still stuck to his hips. I added SETs of my palm position and elbow position inside my BP_leapController so now when I call those variables they have something in it FYI is this good or bad?

I guess I mentally shortcutted the process in my mind, what I meant by the persona preview is go to your AnimationBlueprint and turn your joints into variables. After having them fed in to the IK you can change any of the default values to your left and you will see what effect that has. This is a very handy way of previewing, debugging and changing your animation setup without adding the complexity of your actual input system.

Here is what I mean:


Also make sure that you IK alpha is set to 1.0 or you will not see any change as the animation setup will still use your animations fully instead of your IK.

Thanks, getnamo! The .dll switch didn't do anything, but I had forgotten to enable images from within the plug-in!

(Also for some reason the billboards were disappearing so I had to route it to a static mesh with a custom material).

It's sweet!

Great In my case the billboard was simply a component I had attached to my character to quickly debug the image output.

[Joe1029]

Hey getnamo I created two sockets behind each elbow and set them to 60 and -60 on the Y axis but that did not fix my characters fondling problem. Am I misunderstanding what you explained?

[zalo]

Thanks, getnamo! The .dll switch didn't do anything, but I had forgotten to enable images from within the plug-in!

(Also for some reason the billboards were disappearing so I had to route it to a static mesh with a custom material).

It's sweet!

[Joe1029]

Ok when you say play around with the values in persona do you mean I go into persona find the location of my characters elbows (lowerhand_l and lowerhand_r) and plug those in as join target location?

I get movement out of my hands but theyre still stuck to his hips. I added SETs of my palm position and elbow position inside my BP_leapController so now when I call those variables they have something in it FYI is this good or bad?

[getnamo]

Hey Getnamo thanks for the advice. So am I correct in using the wrist position for this? I am using an oculus and making this game for oculus so I do have the HMD optimized ticked. I checked out the values for my lowerhand_L but I don't know what you mean by forward direction of the joint target location. Could you explain a little more?

Depending on what you're trying to do, wrist can work, but so can the hand->palm position which is more appropriate for hand position.

In IK setups you need a joint target. Generally these are 2-bone IKs and they behave somewhat like an elbow or knee type joint does on a person. So for the IK setup to behave properly you need your joint target be 'behind' your elbow, so that when the positions cause the bones to bend it will bend towards the direction of the elbow. Play around with the values in persona to find the sweet spot for the joint target.

Oh no, there are no leftover functions in the event graph or anything. It's a fresh new project. It's a new FPS C++ template project. I go into the MyCharacter blueprint and ONLY try to add the LeapEventInterface in implemented interfaces. It saves and compiles fine. But then when I exit the project, and re-open it, I get the above error.

Later, if I try to add even a single LeapEvent or anything to do with the LeapMotion in the EventGraph, it, too, saves and compiles fine. But upon exiting and trying to re-open I get the original error where the editor doesn't even open.

I realize if you can't reproduce the error, there is not much you can do. But hopefully you can help because this is maddening.

I should note the LeapEventInterface isn't being removed by ME. I receive the above error message and the LeapInterface is automatically removed.

Ok found a temporary workaround. Create a fps template, duplicate the MyCharacter in case you mess up in the future (as far as I know the bp is not recoverable after you add the component or interface), now for the fix:

1.Sub-class the Mycharacter in blueprint which by default will make a MyCharacter_Child.
2.Add your leap interface and leap component to this child class.

This will put a layer of indirection to your C++ class and the leap blueprint and it ... works.

I believe this stems from the origins of the plugin which retained a direct link to the Leap header and put all the classes into private implementation. These are still visible in blueprint, but I think C++ cannot include them properly. I will need to rethink how the architecture is built and re-implement it in such a way that the classes can be publicly exposed and included in C++ source. Until then just sub-class your C++ class and use that child to implement leap functions.

[Joe1029]

Hey Getnamo thanks for the advice. So am I correct in using the wrist position for this? I am using an oculus and making this game for oculus so I do have the HMD optimized ticked. I checked out the values for my lowerhand_L but I don't know what you mean by forward direction of the joint target location. Could you explain a little more?

[getnamo]

Gah, this plug-in so so cool, but I can't get my favorite part to work!

Whenever I try to access an image, this appears for every frame in the output:


I have image support enabled of course; all of my Unity apps work. Disabling Image support has no effect.

All non-image functions seem to work fine (fingertip positions, etc), but I was really hoping to make a kick-*** passthrough demo...

You need to make sure you're on the 2.1.5 sdk and you enable it from the blueprint via LeapController->enable image support. Typically you'd call this at begin play or another init type function. This follows how the Leap API expects this to work, probably an optimization to not use the extra bandwidth when not needed.

[getnamo]

Oh no, there are no leftover functions in the event graph or anything. It's a fresh new project. It's a new FPS C++ template project. I go into the MyCharacter blueprint and ONLY try to add the LeapEventInterface in implemented interfaces. It saves and compiles fine. But then when I exit the project, and re-open it, I get the above error.

Later, if I try to add even a single LeapEvent or anything to do with the LeapMotion in the EventGraph, it, too, saves and compiles fine. But upon exiting and trying to re-open I get the original error where the editor doesn't even open.

I realize if you can't reproduce the error, there is not much you can do. But hopefully you can help because this is maddening.

I should note the LeapEventInterface isn't being removed by ME. I receive the above error message and the LeapInterface is automatically removed.

I was able to reproduce it following what you said, it seems to be related to C++ code as a base vs a blueprint one, will check around what causes this. This shouldn't be happening and it appears this type of problem is more widespread than just this plugin: https://answers.unrealengine.com/que...reeeditor.html.

Will let you know if I find a fix.

[zalo]

Gah, this plug-in so so cool, but I can't get my favorite part to work!

Whenever I try to access an image, this appears for every frame in the output:

[2014.10.21-18.20.34:842][101]LogClass:Error: Warning! Leap Image SDK access is denied, please enable image support from the Leap Controller.

I have image support enabled of course; all of my Unity apps work. Disabling Image support has no effect.

All non-image functions seem to work fine (fingertip positions, etc), but I was really hoping to make a kick-*** passthrough demo...

EDIT: Hmm, it seems I'm using a slightly newer version (2.1.5+22699)... drat they don't have the old one online.

Maybe updating the .dll will help...