[Plugin] Leap Motion - Event Driven

[timsuitcase]

Hi guys!

So I’m really excited! I’ve got the plugin working in my project. In the sense that I’ve loaded the actor and I can see the hands in VR.

However, the next step: letting the hand interact with my products (grab), is something I’m struggling with. The tutorials I found so far on getting the hands in VR were pretty easy. But I’m sort of missing something to let the hands interact…

Could somebody help me? Of course, compensation is available, because as we like to say in Dutch: only the sun rises for free (so everything else should be paid).

All the best,

Tim

[jakwarne]

Hi everyone,

I had a project in 4.18 using the v2 Leap Motion framework . Now that that we’re on 4.21 and v4 framework, does that mean that I have to rebuild all my blueprints related to the Leap Motion?

I’m asking because it seems like a lot of things have been changed with the plugin. Did anyone else have the same problem?

Thanks,

[getnamo]

@jakwarne

Hi everyone,

I had a project in 4.18 using the v2 Leap Motion framework . Now that that we’re on 4.21 and v4 framework, does that mean that I have to rebuild all my blueprints related to the Leap Motion?

I’m asking because it seems like a lot of things have been changed with the plugin. Did anyone else have the same problem?

Thanks,

The v3 plugin (using v4 tracking) is a complete rewrite so it has fully breaking API. That said it should be easier to get things running and you will find it much more performant with lower latency. It is not expected to have significant breaking API like this in the future.

Check out GitHub - leapmotion/LeapUnrealModules: Leap Motion Unreal modules and example content. for some basic blueprint examples.

@timsuitcase

Hi guys!

So I’m really excited! I’ve got the plugin working in my project. In the sense that I’ve loaded the actor and I can see the hands in VR.

However, the next step: letting the hand interact with my products (grab), is something I’m struggling with. The tutorials I found so far on getting the hands in VR were pretty easy. But I’m sort of missing something to let the hands interact…

Could somebody help me? Of course, compensation is available, because as we like to say in Dutch: only the sun rises for free (so everything else should be paid).

All the best,

Tim

There is a basic pickup/drop button press example found here: GitHub - leapmotion/LeapUnrealModules: Leap Motion Unreal modules and example content. try out the map and explore the blueprints to see how it all comes together.

[jakwarne]

So… I have updated to the new plugin and SDK, thanks for the work @getnamo. What is the situation now with gestures such as swipe? Do you know of an effective way to do that?

Thanks

[getnamo]

@jakwarne

So… I have updated to the new plugin and SDK, thanks for the work @getnamo. What is the situation now with gestures such as swipe? Do you know of an effective way to do that?

Thanks

The inbuilt gesture detections have been deprecated, see this issue for detail: https://github.com/leapmotion/LeapUnreal/issues/11. It isn’t too hard to make swipe gestures (some basic concepts are discussed in the issue) but if there is sufficient demand, an example could be added in the LeapUnrealModules implementing basic gesture recognizers.

[Shy_ELC]

Hi Jan!

We’ve been playing with the Leap Motion plugin for UE4 for a few weeks now and found it quite informative, thank you for putting it together!

One thing we’d like to know is how to go about rebuilding our own customised version of the plugin. Our use case involves using specific gestures which are not currently supported by the plugin v3.2, and we would also like to tweak the behaviour of the existing gestures to work more seamlessly with our intended UX. To this end we went digging deep into the plugin code to see how we might go about doing this.

However, we have found that any attempt to rebuild the plugin from .uproject->Open or from Visual Studio->Build fails with a list of

cannot find *.dll

errors, even if we try to build from unmodified source. We are able to reboot the project by using the pre-built assemblies, but this doesn’t give us the ability to modify functionality that we’re looking for.

This is blocking our ability to really experiment with the Leap Motion in a meaningful way.

Is this intended behaviour, or does it seem obvious that we should be approaching it a different way?

Any assistance would be appreciated!

[smbv1]

Hello, I would like to develop a desktop game with LeapMotion,
can you please help to answer a few questions:

1. Can I start developing with 4.21.1 out of the box or I need to download a special plugin and install it to 4.21.1 Engine?
2. Where do I download some initial pack? Altough I’ve made many VR games, I would like to start from desktop-demo.

Can you please guide me with these questions and show me how to use it with Blueprints, thank you very much!

[getnamo]

@Shy_ELC

Hi Jan!

We’ve been playing with the Leap Motion plugin for UE4 for a few weeks now and found it quite informative, thank you for putting it together!

One thing we’d like to know is how to go about rebuilding our own customised version of the plugin. Our use case involves using specific gestures which are not currently supported by the plugin v3.2, and we would also like to tweak the behaviour of the existing gestures to work more seamlessly with our intended UX. To this end we went digging deep into the plugin code to see how we might go about doing this.

However, we have found that any attempt to rebuild the plugin from .uproject->Open or from Visual Studio->Build fails with a list of

cannot find *.dll

errors, even if we try to build from unmodified source. We are able to reboot the project by using the pre-built assemblies, but this doesn’t give us the ability to modify functionality that we’re looking for.

This is blocking our ability to really experiment with the Leap Motion in a meaningful way.

Is this intended behaviour, or does it seem obvious that we should be approaching it a different way?

Any assistance would be appreciated!

The current plugin found at GitHub - leapmotion/LeapUnreal: Leap Motion SDK for the Unreal Engine. should work both as project plugin and engine plugin as it detects it’s environment. The general way to rebuild the plugin is to include it in e.g. a ue4 C++ project under plugins and regenerating your .sln then recompiling, the plugin will recompile with it.

The same process is used for engine plugins except that you replace the plugin found under e.g.

UE_4.21/Engine/Plugins/Runtime/LeapMotion

and you copy your third party includes to

UE_4.21/Engine/Source/ThirdParty/Leap

and dlls to

UE_4.21/Engine/Binaries/ThirdParty/LeapMotion

I’m hoping we will have an updated leap motion plugin for 4.22 which should eliminate the engine plugin workaround, but can give no guarantees at this time.

[getnamo]

@smbv1

Hello, I would like to develop a desktop game with LeapMotion,
can you please help to answer a few questions:

1. Can I start developing with 4.21.1 out of the box or I need to download a special plugin and install it to 4.21.1 Engine?
2. Where do I download some initial pack? Altough I’ve made many VR games, I would like to start from desktop-demo.

Can you please guide me with these questions and show me how to use it with Blueprints, thank you very much!

1. Get the latest plugin at Releases · leapmotion/LeapUnreal · GitHub and follow instructions found here: GitHub - leapmotion/LeapUnreal: Leap Motion SDK for the Unreal Engine.

2. For desktop use see this heading GitHub - leapmotion/LeapUnreal: Leap Motion SDK for the Unreal Engine.

[jakwarne]

Hi all,

How do I get the Grab input to work using VRPickupDrop template with LeapMode=Desktop?

Regards

[getnamo]

@jakwarne

Hi all,

How do I get the Grab input to work using VRPickupDrop template with LeapMode=Desktop?

Regards

It should work as is with a single call to SetLeapMode with LeapMode set to *Leap_Mode_Desktop *e.g. somewhere in begin play. The function is global so it is available in any event graph just right click and start typing set leap mode. See GitHub - leapmotion/LeapUnreal: Leap Motion SDK for the Unreal Engine. for details.

Note that you may need to change the pawn used in that example to have desktop orientation (facing up), see the LeapUnreal/LeapDesktopActor.uasset at master · leapmotion/LeapUnreal · GitHub blueprint in your editor for an example of how the childactor is set up.

[KimLaughton]

I’ve just been playing with the auto mapper and while it’s working I’m having a few issues - I’m sure they’re fairly easy to solve but I’m not sure where to start.

I’m trying to map just the right hand of a model (the model itself is a complete skeleton). Firstly the mapping is quite distorted, especially the thumb which seems to be flipped around. How should I be adjusting this exactly? I couldn’t figure it out from the documentation. Secondly the hand sometimes flips - from left to right. Is there any way to let the plugin know that I’m only ever dealing with a right hand so that this doesn’t happen? The project will form part of a public interactive installation, so the more solid the results the better.

Thanks!

[getnamo]

@KimLaughton

I’ve just been playing with the auto mapper and while it’s working I’m having a few issues - I’m sure they’re fairly easy to solve but I’m not sure where to start.

I’m trying to map just the right hand of a model (the model itself is a complete skeleton). Firstly the mapping is quite distorted, especially the thumb which seems to be flipped around. How should I be adjusting this exactly? I couldn’t figure it out from the documentation. Secondly the hand sometimes flips - from left to right. Is there any way to let the plugin know that I’m only ever dealing with a right hand so that this doesn’t happen? The project will form part of a public interactive installation, so the more solid the results the better.

Thanks!

Inside your anim instance class defaults, Ensure automap target is HAND RIGHT for your case. Also below that section is a structure called mapped bone list open it up you can see Pre-base rotation and offset transform, these will apply to the entire mapped bone list. Pre-base rotation will apply rotation before your values (akin to changing the basis of your input) where as offset transform applies after your input has been fed. If you’re moving your hand/fingers and they bend in a wrong axis e.g. they yaw when they should pitch, you will need to adjust the pre-base rotation, typically in some combination of 90 degrees. These will typically be different for each hand if your mesh is rigged like animators typically do with positive rotation closing your fist. In the low poly hands it uses a pre base of -90 for left and 90,0 180 for right.

That said it looks like your thumb might be in a different space than the rest of your hand and you may need to remove it from the auto mapping list and feed the rigging directly, or go back and adjust your model in your software. If it’s just a fixed offset difference however, you can just add a standard Transform (Modify) Bone node and adjust the offset there after the Modify Mapped Bones node.

If the auto-rigging is giving you a problem, checkout the discussion here for basic details on direct rigging:
https://github.com/leapmotion/LeapUnreal/issues/35

Regarding the flipping, it might be occurring due to hand chirality detection, might need a video of the issue to know more.

[KimLaughton]

@getnamo

Inside your anim instance class defaults, Ensure automap target is HAND RIGHT for your case. Also below that section is a structure called mapped bone list open it up you can see Pre-base rotation and offset transform, these will apply to the entire mapped bone list. Pre-base rotation will apply rotation before your values (akin to changing the basis of your input) where as offset transform applies after your input has been fed. If you’re moving your hand/fingers and they bend in a wrong axis e.g. they yaw when they should pitch, you will need to adjust the pre-base rotation, typically in some combination of 90 degrees. These will typically be different for each hand if your mesh is rigged like animators typically do with positive rotation closing your fist. In the low poly hands it uses a pre base of -90 for left and 90,0 180 for right.

That said it looks like your thumb might be in a different space than the rest of your hand and you may need to remove it from the auto mapping list and feed the rigging directly, or go back and adjust your model in your software. If it’s just a fixed offset difference however, you can just add a standard Transform (Modify) Bone node and adjust the offset there after the Modify Mapped Bones node.

If the auto-rigging is giving you a problem, checkout the discussion here for basic details on direct rigging:
Make custom rigging example video · Issue #35 · leapmotion/LeapUnreal · GitHub

Regarding the flipping, it might be occurring due to hand chirality detection, might need a video of the issue to know more.

Thanks so much for this info! I’ve fixed the thumb - I’m not sure what was causing it, but restarted from scratch and it’s fine now. I have the right hand automap target and it’s now mostly working fine but everything is a little off.

vid here: Screen capture - dc72381e293d8c17aada694da4d3ca8f - Gyazo

At the moment it feels like the proximal bones aren’t moving at all and all the others look v strange (maybe because of this?)

The mapping looks ok to me. Maybe I should give up with this model and use something that’s based on the epic skeleton - is that likely to give better results?

As far as the flipping is concerned it seems to be happening in the Orion software - sometimes it’s fine and completely solid, other times it there is flipping I guess as you say to due to the chirality detection. I’m not sure quite why it occurs, the environment and location of the sensor is always pretty much the same and the lens is clean. vid here Screen capture - 6572b1d4d08805fd898de4646ed31f9a - Gyazo

[getnamo]

@KimLaughton

Thanks so much for this info! I’ve fixed the thumb - I’m not sure what was causing it, but restarted from scratch and it’s fine now. I have the right hand automap target and it’s now mostly working fine but everything is a little off.

vid here: https://gyazo.com/dc72381e293d8c17aada694da4d3ca8f

At the moment it feels like the proximal bones aren’t moving at all and all the others look v strange (maybe because of this?)

The mapping looks ok to me. Maybe I should give up with this model and use something that’s based on the epic skeleton - is that likely to give better results?

As far as the flipping is concerned it seems to be happening in the Orion software - sometimes it’s fine and completely solid, other times it there is flipping I guess as you say to due to the chirality detection. I’m not sure quite why it occurs, the environment and location of the sensor is always pretty much the same and the lens is clean. vid here https://gyazo.com/6572b1d4d08805fd898de4646ed31f9a

It looks like your hand is deforming to fit the tracked hand data using *Should Deform Mesh *option found here (https://github.com/leapmotion/leapunreal#deformation)

https://i.imgur.com/LhIu3JQ.gif

For this to work your model should have it’s bones placed/closely weighted as the low poly hand does so that the hand correctly stretches to fit the user’s real hand data. If you don’t mind some tip misalignment or your model doesn’t support deformation (e.g. ue4 mannequin model) then untick this option and it will apply rotations only to the hand which works far more universally.

For hand chirality, are you using this attached to a VR headset or is it e.g. on a table facing up? If it’s the latter, you’ll need to call SetLeapMode (https://github.com/leapmotion/leapunreal#set-leap-mode) in e.g. your begin play and set it to LEAP_MODE_DESKTOP this should ensure the tracking software expects the correct chirality based on it’s orientation. The example LeapDesktopActor has this call, open the blueprint to see where the calls are done.

[KimLaughton]

Actually I already turned off the deformation, so I guess it’s not that. Not quite sure why it’s so far from where it ought to be because the hand rig looks fairly conventional to me. There were some UE hands/arms for Leap Motion around before, but they don’t seem to be included in the current plugin. Are they still available and is the rigging still suitable for the latest version of the software?

Setting it to desktop mode seems to have solved the flipping thanks! Also great to have the option for more smooth input, it was a bit jittery before.

@getnamo

It looks like your hand is deforming to fit the tracked hand data using *Should Deform Mesh *option found here (GitHub - leapmotion/LeapUnreal: Leap Motion SDK for the Unreal Engine.)

https://i.imgur.com/LhIu3JQ.gif

For this to work your model should have it’s bones placed/closely weighted as the low poly hand does so that the hand correctly stretches to fit the user’s real hand data. If you don’t mind some tip misalignment or your model doesn’t support deformation (e.g. ue4 mannequin model) then untick this option and it will apply rotations only to the hand which works far more universally.

For hand chirality, are you using this attached to a VR headset or is it e.g. on a table facing up? If it’s the latter, you’ll need to call SetLeapMode (GitHub - leapmotion/LeapUnreal: Leap Motion SDK for the Unreal Engine.) in e.g. your begin play and set it to LEAP_MODE_DESKTOP this should ensure the tracking software expects the correct chirality based on it’s orientation. The example LeapDesktopActor has this call, open the blueprint to see where the calls are done.

[getnamo]

@KimLaughton

Actually I already turned off the deformation, so I guess it’s not that. Not quite sure why it’s so far from where it ought to be because the hand rig looks fairly conventional to me. There were some UE hands/arms for Leap Motion around before, but they don’t seem to be included in the current plugin. Are they still available and is the rigging still suitable for the latest version of the software?

Setting it to desktop mode seems to have solved the flipping thanks! Also great to have the option for more smooth input, it was a bit jittery before.

Check out GitHub - leapmotion/LeapUnrealModules: Leap Motion Unreal modules and example content. for rigging examples (download project, drag and drop matching unreal engine plugin and open the CustomRigHello.umap to see various assets rigged)

[KimLaughton]

@getnamo

Check out GitHub - leapmotion/LeapUnrealModules: Leap Motion Unreal modules and example content. for rigging examples (download project, drag and drop matching unreal engine plugin and open the CustomRigHello.umap to see various assets rigged)

Thanks! Sorry, missed that before - very handy.

Afraid it looks like the flipping still hasn’t gone away totally - for a while it seems really solid and then sometimes it comes back and sticks around. This happens both in UE and the visualizer, as seen here:

Are there any things that could lead to this do you know? Should I try getting another sensor or might it be something to do with the conditions it’s used in? It seems similar to the issue here:

Hand left-right recognition

If there were a way to let it know that it would only ever be looking at a right hand then this would presumably solve it - but I don’t know if there is?

[lumbabumba]

Hi,

ty for your plugin. I am trying to get the camera image of the LeapMotion Controller to do some processing with OpenCV.
I have added LeapMotion to my modules and I can compile.

But I found this here

 [TABLE]

/** Enable or disable image streaming from the device*/
 		

 
 			//UPROPERTY(BlueprintReadWrite, Category = "Leap Options") //disabled for now, current version doesn't actually support image streaming
 		

 
 			bool bEnableImageStreaming;
 		

 

Is there any way to get images or frames to use OpenCV on it?

Ty, sry for bad english.

[getnamo]

@lumbabumba

Hi,

ty for your plugin. I am trying to get the camera image of the LeapMotion Controller to do some processing with OpenCV.
I have added LeapMotion to my modules and I can compile.

But I found this here…

Is there any way to get images or frames to use OpenCV on it?

Ty, sry for bad english.

Answered here: Get Images or Frames · Issue #41 · leapmotion/LeapUnreal · GitHub. Use SetLeapPolicy to enable raw image streaming, then subscribe to the OnImageEvent on your leap component.