[Plugin] Leap Motion - Event Driven

[getnamo]

Hi [MENTION=548]getnamo[/MENTION] - thanks for an amazing plugin!

I just wanted to mention here that I'm also trying to get network replicated Leap Motion hands working. It seems myself and several others are in the same boat. I've created an AnswerHub question and an issue on the GitHub page for the leap plugin.
[MENTION=425607]Joris Vial[/MENTION], [MENTION=491888]SP2016[/MENTION], [MENTION=102913]JLBXB3[/MENTION] - have any of you made any progress on this problem?

Thanks,

I've looked into this and the transition to body state in the next dev release should enable replication correctly. Stay tuned

[getnamo]

Please help me.
A few days of suffering...

I try to attach the object into the socket.

https://answers.unrealengine.com/que...to-socket.html

Please check your question for the answer, I've added a reply.

Greetings all.

I'm working on a Leap Motion project where as much game logic as possible is being kept in C++. The documentation on github implies that the same setup logic in blueprint should be applicable in C++, but I've hit several stumbling blocks trying to do so. I'd just like to check that I'm not missing anything obvious in my approach.

First off, when using the in engine plugin included with 4.12.5, trying to access any of the functions of the classes in the AnimBody folder results in LNK2019 errors. After some digging, I realized this was because said classes were all missing the LEAPMOTION_API specifier. Adding the specifier back in resolved the LNK2019 errors, but I've had problems instantiating them in C++. I've tried CreateDefaultSubobject and NewObject to make AnimHand objects, but the internal AnimFingers/AnimBones always end up null. Is this set of classes just not setup to work in C++? Or am I doing something wrong?

Also, I wanted to make use of the LeapEventInterface, but they're all specified as BlueprintImplementableEvents. This means I can't directly inherit the class and provide a C++ implementation, correct? Or am I missing something?

This is a good catch, they should be BlueprintNativeEvent instead, which will allow your interface object respond implement the events in e.g. this format:

virtual void LeapRightHandMoved_Implementation(ULeapHand* Hand) override;

for each event.

I've added this as a github issue for now with a branch; will test this on the next plugin pass.

Hey there getnamo, thanks very much for the hard work you've put into this plugin. Echoing what some others have posted, I have had similar collision issues following your Jenga tutorial with the FloatingHandsCharacter - I can't seem to detect on hit events in mesh to pick up's blueprint, where we check if it implements the interface - I made this work with a block component in the FloatingHandsCharacter BP (made a "stick" and got attaching a pickup cube to the "stick" to work by toggling grab with a key press) - this is also for a university project, I've tried setting the collision in the meshes (LeftHandMesh and RightHandMesh) in LeapRiggedEchoHandsActor to PhysicsActor but that doesn't work. The hands are grabbing but they are not being detected as touching the cube. If anyone has a solution to this it would be greatly appreciated!

Check your character settings, they may be overriding the ability of your meshes to collide. Also keep in mind that the jenga video is just one example of how to get something like that working and should not be considered the definite way.

[4aries17]

Please help me.
A few days of suffering...

I try to attach the object into the socket.

https://answers.unrealengine.com/que...to-socket.html

[catonice]

Hey there getnamo, thanks very much for the hard work you've put into this plugin. Echoing what some others have posted, I have had similar collision issues following your Jenga tutorial with the FloatingHandsCharacter - I can't seem to detect on hit events in mesh to pick up's blueprint, where we check if it implements the interface - I made this work with a block component in the FloatingHandsCharacter BP (made a "stick" and got attaching a pickup cube to the "stick" to work by toggling grab with a key press) - this is also for a university project, I've tried setting the collision in the meshes (LeftHandMesh and RightHandMesh) in LeapRiggedEchoHandsActor to PhysicsActor but that doesn't work. The hands are grabbing but they are not being detected as touching the cube. If anyone has a solution to this it would be greatly appreciated!

[jlin98]

Greetings all.

I'm working on a Leap Motion project where as much game logic as possible is being kept in C++. The documentation on github implies that the same setup logic in blueprint should be applicable in C++, but I've hit several stumbling blocks trying to do so. I'd just like to check that I'm not missing anything obvious in my approach.

First off, when using the in engine plugin included with 4.12.5, trying to access any of the functions of the classes in the AnimBody folder results in LNK2019 errors. After some digging, I realized this was because said classes were all missing the LEAPMOTION_API specifier. Adding the specifier back in resolved the LNK2019 errors, but I've had problems instantiating them in C++. I've tried CreateDefaultSubobject and NewObject to make AnimHand objects, but the internal AnimFingers/AnimBones always end up null. Is this set of classes just not setup to work in C++? Or am I doing something wrong?

Also, I wanted to make use of the LeapEventInterface, but they're all specified as BlueprintImplementableEvents. This means I can't directly inherit the class and provide a C++ implementation, correct? Or am I missing something?

[SP2016]

Hi 6ead2ebf, I tried to serialize the anim body object in json and send it to another client but i didn't succed.
So i simply sent shoulder, elbow, wrist and all fingers orientations taken directly from the LeapRiggedCharacter over the network in json and applied them to the bones of the mesh on another client.

I just needed the mesh on the second client follow the movements of the one on the first client(the one with the leap).
I know this is not the correct solution nor it isn't elegant but it works.

[6ead2ebf]

Hi [MENTION=548]getnamo[/MENTION] - thanks for an amazing plugin!

I just wanted to mention here that I'm also trying to get network replicated Leap Motion hands working. It seems myself and several others are in the same boat. I've created an AnswerHub question and an issue on the GitHub page for the leap plugin.
[MENTION=425607]Joris Vial[/MENTION], [MENTION=491888]SP2016[/MENTION], [MENTION=102913]JLBXB3[/MENTION] - have any of you made any progress on this problem?

Thanks,

[getnamo]

Thank you for the answer!
I'm already using the hand moved event to get the data that i send from the client to the server, my problem is that when another client receives the hand's data of the first client, i can't find a way to move the hands of the mesh representing the other player(my application is a sort of virtual room with 2 people that can interact in some way using their hands and moving them with the leap).

On the second client i receive correctly the data regarding other person's hands but i can't move the mesh's hands correctly, i saw that you used some sort of orientation variables to move the hands, can i use instead elbow, wrist and fingers components positions(read from the leap hand moved event on the first client) to move the elbow, wrist and finger components on the second client? When i tried to do this i get some strange results like deformed fingers and so on.

Which is the correct way of binding leap data received from a server(so i can't use event hands moved) to a BasicRiggedLeapCharacter?

Your help is very appreciated.
Thank you!

The animation is setup is to get close to 1:1 representation of your hand/finger positions. The way you can do this with a skeletal mesh is that you IK to the elbow/wrist position which the leap motion estimates, and then you use the orientations of everything else which will put the fingers in the correct form without distortion. The fingers might not match your real finger length but the relation of the fingers in the mesh will line up correctly for nearly any type of skeletal mesh hand.

If you wish to use this animation setup, then the correct way of updating the position from an external source is to update the Anim body which the animation blueprint uses.

AnimHand has a convenience function written in C++ which grabs the information directly from a leap hand (https://github.com/getnamo/leap-ue4/...nimHand.h#L128) and this is called in blueprint (in leap connector I believe). What you need to do is to set the Anim body data similarly to https://github.com/getnamo/leap-ue4/...imHand.cpp#L91 (this can be done in blueprint), all the setting functions are publically accessible. Another way would be if you could serialize the leaphand data and pipe that over the network, but that might be a bit harder to do.

Optionally setup your own skeletal mesh animation rig which uses raw data you pipe in a different way.

Finally a last option is to fix the blueprints to work correctly in UE's networking system through replication, which is something [MENTION=425607]Joris Vial[/MENTION] seems to be working through.

[SP2016]

AnimBody is used to create a separation from leap specific input to the animation of the character mesh. But since you want to use the leap motion data directly you may find it easier to do the custom bp approach (attaching a leap controller to any blueprint of your choice and adding an interface to that class)
https://github.com/getnamo/leap-ue4#...nt-quick-setup

and then just push the data you get from the event on hand moved:
https://github.com/getnamo/leap-ue4#...--event-driven

which is much closer to the raw data you get from leap (albeit converted to UE4 space and units).

Hope that helps!

Thank you for the answer!
I'm already using the hand moved event to get the data that i send from the client to the server, my problem is that when another client receives the hand's data of the first client, i can't find a way to move the hands of the mesh representing the other player(my application is a sort of virtual room with 2 people that can interact in some way using their hands and moving them with the leap).

On the second client i receive correctly the data regarding other person's hands but i can't move the mesh's hands correctly, i saw that you used some sort of orientation variables to move the hands, can i use instead elbow, wrist and fingers components positions(read from the leap hand moved event on the first client) to move the elbow, wrist and finger components on the second client? When i tried to do this i get some strange results like deformed fingers and so on.

Which is the correct way of binding leap data received from a server(so i can't use event hands moved) to a BasicRiggedLeapCharacter?

Your help is very appreciated.
Thank you!

[getnamo]

Hi getnamo and everyone! This plugin is great and i'm learning to use it

I'm using the plugin to read all hand's data from the first client and send them to a custom nodejs server. On the second client i'm trying to position the hand and its fingers according to the infos received from the server and i can't understand how to accomplish this last thing.


In LeapBasicRiggedCharacter's anim BP (BasicCharacter_AnimBlueprint), in the SetLocalVariables function all fingers orientations are set(e.g. Left Middle 1Orientation), for what i know the Leap API gives you the positions of finger bones, i can't understand why in the animBP orientations are used, can you explain me this?

Also should i apply received values to set directly variables like 'Left Middle 1Orientaton' or can i simply use "Transform Bone" functions?
I see that there an intermediate step in your plugin that is the AnimBody object, should i set received values in it?

Sorry for the many questions but i'm getting crazy over this.
I thank you for any help you or anyone can give me.

AnimBody is used to create a separation from leap specific input to the animation of the character mesh. But since you want to use the leap motion data directly you may find it easier to do the custom bp approach (attaching a leap controller to any blueprint of your choice and adding an interface to that class)
https://github.com/getnamo/leap-ue4#...nt-quick-setup

and then just push the data you get from the event on hand moved:
https://github.com/getnamo/leap-ue4#...--event-driven

which is much closer to the raw data you get from leap (albeit converted to UE4 space and units).

Hope that helps!

[SP2016]

Hi getnamo and everyone! This plugin is great and i'm learning to use it

I'm using the plugin to read all hand's data from the first client and send them to a custom nodejs server. On the second client i'm trying to position the hand and its fingers according to the infos received from the server and i can't understand how to accomplish this last thing.


In LeapBasicRiggedCharacter's anim BP (BasicCharacter_AnimBlueprint), in the SetLocalVariables function all fingers orientations are set(e.g. Left Middle 1Orientation), for what i know the Leap API gives you the positions of finger bones, i can't understand why in the animBP orientations are used, can you explain me this?

Also should i apply received values to set directly variables like 'Left Middle 1Orientaton' or can i simply use "Transform Bone" functions?
I see that there an intermediate step in your plugin that is the AnimBody object, should i set received values in it?

Sorry for the many questions but i'm getting crazy over this.
I thank you for any help you or anyone can give me.

[getnamo]

Hi getnamo and hi everyone.

First, thank you for this amazing plugin. I've tried it out and it works very well out of the box ! This will open new possibilities with VR and I can't wait to see future project using it.

I'm currently working on a networked simulation and I'm trying to figure out how to replicate hands through the network.
I think the plugin is currently not setup to do so (correct me if I'm wrong), so I started to modify a bit the plugin blueprints.

I've managed to replicates a LeapRiggedEchoHandsActor and its sub components/actor but I'm stuck at animating it. For now, the leap motion client implements the HandMoved event and call a server RPC. The issue is that on the server side the LeapHand reference is not valid so I guess this one is not replicated. To do so I have to start modifying the C++ plugin code and this may be a bit harder than expected.

So before I run into this I would like to know if someone already managed to replicated the hands through the network.

Best regards.

Great investigation and I'm looking forward to what solution you come up with. Make sure to make a pull request when you get it working!


On a side note has anyone else had this issue: https://github.com/getnamo/leap-ue4/issues/12 in packaged games? If so what version of the plugin/editor did you see this?

[NymKappa]

Hi getnamo and hi everyone.

First, thank you for this amazing plugin. I've tried it out and it works very well out of the box ! This will open new possibilities with VR and I can't wait to see future project using it.

I'm currently working on a networked simulation and I'm trying to figure out how to replicate hands through the network.
I think the plugin is currently not setup to do so (correct me if I'm wrong), so I started to modify a bit the plugin blueprints.

I've managed to replicates a LeapRiggedEchoHandsActor and its sub components/actor but I'm stuck at animating it. For now, the leap motion client implements the HandMoved event and call a server RPC. The issue is that on the server side the LeapHand reference is not valid so I guess this one is not replicated. To do so I have to start modifying the C++ plugin code and this may be a bit harder than expected.

So before I run into this I would like to know if someone already managed to replicated the hands through the network.

Best regards.

[Andre Jones]

How to detect it my Palm is UP or DOWN and if it is Left or Right Hand?

[getnamo]

Is it impossible to work with the Unreal Engine plugin without a VR system? I have 4.12.4 and the latest version of Desktop V2 (and recently tried Orion too), but when I override the blueprint to be the one with the FloatingHands and hit play, nothing happens. I can still move with my mouse but no input is being read from my Leap Motion.

Desktop mode should still work. Just note that if you do not use VR mode it expects the leap motion on your desk facing up. Please see https://github.com/getnamo/leap-ue4#...n-default-mode for simple note on how to adjust your desktop offset.

You should be able to use the same technique, but you may have to get the child actor which holds the raw hands themselves for collision. E.g. in rigged floating hands you have two skeletal meshes, which may need collision checks for the interface to work. Let me know how this works out so I can add the complete solution to documentation.

Hi Getnamo, I have the same issue. Any chance you could give more details on how to get the child actor? Thanks!

I'll try to find some time to show an example of this in blueprints, stay tuned.

When I downloaded the plugin, there is not LeapCollisionCharacter or VRPlayerController in the menu for selecting gamemode. Im using UE 12.3. Have you removed this from the latest release? If so, why isn't the documentation updated?

Documentation has been updated and references to LeapCollisionCharacter (other characters have their collision on by default) and VRPlayerController (no longer needed) have been removed. Despite this being the official leap motion plugin, it's still a community plugin in that it's a side project I do in my own free time. This means resources may not be always available and it relies on people such as yourself to point out and make pull requests where possible