[Plugin] Leap Motion - Event Driven

[zalo]

It looks like your leap is in robust mode's half-vertical-resolution state in that last gif.

If you add the line

"robust_mode_enabled": false,

to your config.json in your ProgramData->Leap Motion folder, it should avoid this...

[getnamo]

Hey getnamo,

We just want to thank you for your plugin. Seems to work excellent and it's just incredibly easy to integrate.

In case you didn't know, we're developing a soft-body physics engine plugin for Unreal Engine 4, check out our thread here: https://forums.unrealengine.com/show...lug-in-for-UE4

So our leap motion devices came in today and we really couldn't wait to try and see how it integrated with our own soft-body physics engine, so we threw together a quick showcase using your plugin. Pretty surprised to see how easy it all was Here's the result:

...

Keep up the good work!

Love this frog video, awesome work!

Is there anyway to make the Debug Sphere to trigger with trigger box?
(I'm quite new and don't know how to interact with 3D models in the blueprint though, so I use debug sphere instead.)

Your best bet would be to attach a capsule or box to the interaction location and that for the trigger interaction. You can also add components to the convenience rigged content bones or alternatively you can instead use the official plugin which contains rigged content with physics. Finally this plugin will get some physics convenience components added to it eventually, but that may be some time.


Documentation for 0.9

How to use the convenience rigged passthrough character:
1) Select the LeapPassthroughCharacter as your pawn and VRPlayerController as your controller

2) Use the included gesture to transition from VR to AR! (Optionally make your own method, see below for details)





This updates the convenience content relationships to the following


What this setup gives you in flexibility is that you have many entry points from which to modify any aspect of the rigging. E.g. If you wish to change post process material being used you would change it in the PPChanger blueprint while still retaining every other functionality. Or if you want to adjust the default mode offset (where your leap is in relation to you when place on the table) you would modify the LeapAnimBodyConnector component which specifies its location. If you want to use your own skeletal mesh, simply replace the mesh, but retain the anim blueprint (use animation retargeting if you're using different skeletons). In fact the rigging setup doesn't even need the leap motion to work, any input which would change the AnimBody skeleton to whatever pose you wish to attain would have the same rigged behavior. The setup is the beginning part of a BodyInput plugin which will support a similar setup for various VR inputs in the future.

Attaching Passthrough to your own Character
If you attach a LeapBodyConnectorWithPassthrough actor as a child actor you will automatically get an AnimBody that will sync with your leap movements. To enable passthrough you call this function in some initializer such as a BeginPlay. You have the option of disabling the pinch + swipe up/down gesture for transitions.



If you wish to make your own transitions you can call ShowPassthrough on the connector.



which is simply a float parameter change in the post process material.



If you want to see how this has been implemented, browse the LeapAnimBodyConnectorWithPassthrough blueprint.



In there you will notice we obtain 4 image pointers, one for each distortion and one for each raw image. These are then fed to a PostProcessChanger (PPChanger) convenience component which handles adding a custom blendable with the correct links to the character camera.



this then uses the following post process material to render the images appearing correct for the DK2, compensating for both fish-eye lens distortion in the leap cameras and the rift warp.



This would also be where you would duplicate and modify the material to produce interesting effects such as depth-based blending or custom depth drawing (3d element overlay in real world)




The plugin still supports all the earlier feature sets so if you wish to build your own bind, you have all the tools available.

Let me know if this has been helpful, if you would do things differently, or general questions/feedback. Hope you guys like it!

[Tony Tong]

Is there anyway to make the Debug Sphere to trigger with trigger box?
(I'm quite new and don't know how to interact with 3D models in the blueprint though, so I use debug sphere instead.)

[getnamo]

Update to 0.9
-Binaries updated to UE4.7
-Added properly HMD warped image passthrough support through both
materials and a convenience character (LeapPassthrougCharacter)
-128bit texture support resolved for Distortion textures. Use Distortion
for proper float data, use DistortioUE to visualize the texture for
debug purposes.
-Optional content moved into plugin, delete if you don't use it and wish
to save space
-Rigging update to be more loosely coupled. Character anim blueprint
only depends on AnimBody obtained through a simple and safe
AnimBodyInterface
-Rigged character finger rotation bug fixed
-Updated leap bind to 2.2.3, much improved tracking gained.
-Added R->Reset position to rigged characters, change key bind to
reflect own bindings

This post will be updated at a later date with more information/tutorial.
Edit:

Update to 0.9.1
-Cleaned up inheritence structure, LeapAnimBodyActor renamed to
LeapAnimBodyConnector. Added Passthrough sub-class
-LeapPassthroughCharacter now contains the subclass. Rigged characters
contain no passthrough code
-Fixed elbow rotation bug

See my next post below for more documentation on the new passthrough feature.

[seandennis]

I wonder who will be the first to make VR poker/ black jack game with oculus and leap motion. including multi-player

[Tendr Dynamics]

Hey getnamo,

We just want to thank you for your plugin. Seems to work excellent and it's just incredibly easy to integrate.

In case you didn't know, we're developing a soft-body physics engine plugin for Unreal Engine 4, check out our thread here: https://forums.unrealengine.com/show...lug-in-for-UE4

So our leap motion devices came in today and we really couldn't wait to try and see how it integrated with our own soft-body physics engine, so we threw together a quick showcase using your plugin. Pretty surprised to see how easy it all was Here's the result:

Keep up the good work!

[getnamo]

Official Leap Motion Plugin Announced!

I've been looking forward to this release From my interaction with it earlier, you will find a very easy to use plugin!

[BenjaminDSmithy]

Official Leap Motion Plugin Announced!

[knack]

Seems we going to have a UE4 offical plugin from/for LM

News tomorrow

[getnamo]

Hi, I have just a couple of questions. I'm working with a group for a VR project at our college and we bought the Leap Motion hoping to use it as an in between while we waited for our VR gloves to come in (hopefully some time next month or April) I found your plugin and have been trying to adapt it into our project(which is very basic at the moment) I'm able to get everything to work correctly when I first load the plug in into the project. Play in Editor works great, the leap recognizes both hands all individual finger movements etc.

FIRST ISSUE:
However when I try and launch standalone, after the whole cooking process I get an error telling me that Leap.DLL could not be found on my computer. I've checked through the files in the project and the files that came with your plugin as well as Leaps SDK. Leap.dll is all over the place, I even copy pasted it to several other locations, still getting this error every time I try and launch stand alone.

Launching is not supported, use play or play standalone. If you want to distribute it, read the packaging/shipping instructions found on the github readme.

SECOND ISSUE:
After initially loading the Leap plugin into our project and everything working great, I saved everything so as not to lose the setup, as it was working and then closed UE4. When I went to launch the project again later, I get around 430-450 load errors on launch, varying each time I launch the project. I didn't change anything between getting leap to work and closing the project and didn't touch the project between closing it and relaunching it later that same hour.

Attached are a couple of screen shots. One showing the content browser from UE4 which shows all of the Leap plugin folders still in the same location I placed them when I got the plugin to work the first time. The other screen shot is just a small number of the Load errors I'm getting when launching after initially getting leap to work.


Side note...I can delete the copy of the project I made to test the plugin with (and in the process deleting all of the leap motion items) then create another copy, re-add all of the leap stuff to it and it works again but only in Play in Editor, and only that first time. After I save/close/relaunch it load errors again. And the Standalone has never worked.

We are using the latest Leap SDK, with UE 4.6.1

the second issue may be related to this: https://forums.unrealengine.com/show...l=1#post167298
TL;DR: If you're sourcing your classes from C++ you may need to create a middle-man blueprint where you add your leap parts. Still unsure why this happens.

[Tanthalason]

Hi, I have just a couple of questions. I'm working with a group for a VR project at our college and we bought the Leap Motion hoping to use it as an in between while we waited for our VR gloves to come in (hopefully some time next month or April) I found your plugin and have been trying to adapt it into our project(which is very basic at the moment) I'm able to get everything to work correctly when I first load the plug in into the project. Play in Editor works great, the leap recognizes both hands all individual finger movements etc.

FIRST ISSUE:
However when I try and launch standalone, after the whole cooking process I get an error telling me that Leap.DLL could not be found on my computer. I've checked through the files in the project and the files that came with your plugin as well as Leaps SDK. Leap.dll is all over the place, I even copy pasted it to several other locations, still getting this error every time I try and launch stand alone.

SECOND ISSUE:
After initially loading the Leap plugin into our project and everything working great, I saved everything so as not to lose the setup, as it was working and then closed UE4. When I went to launch the project again later, I get around 430-450 load errors on launch, varying each time I launch the project. I didn't change anything between getting leap to work and closing the project and didn't touch the project between closing it and relaunching it later that same hour.

Attached are a couple of screen shots. One showing the content browser from UE4 which shows all of the Leap plugin folders still in the same location I placed them when I got the plugin to work the first time. The other screen shot is just a small number of the Load errors I'm getting when launching after initially getting leap to work.


Side note...I can delete the copy of the project I made to test the plugin with (and in the process deleting all of the leap motion items) then create another copy, re-add all of the leap stuff to it and it works again but only in Play in Editor, and only that first time. After I save/close/relaunch it load errors again. And the Standalone has never worked.

We are using the latest Leap SDK, with UE 4.6.1

[bullale]

Back when I started with the hydra plugin (4.0) there wasn't as much documentation so I wasn't aware of the UE4 delegate system. I was looking for an architecture that would allow me to add event driven, auto-ticking to any class where you would've wanted it in C++. As time went on and I learned more about UE4 it became clear that components offer more flexibility and you could receive events by adding an interface. This allowed each plugin to now extend functionality directly in blueprints rather than C++. I still believe this, along with including convenience characters/actors that can be used for common use cases (e.g. rigged character), is the best approach.

If you can see a way to make your code cleaner by using the UE delegate system, use it. Let us know how it goes!

Thank you for your reply.

One major advantage of the UE delegate system is that you can have more than one listener. I think this could still work with the UActorComponent. It might be possible to have many Actors with LeapController within. Anyway, the documentation is still not quite there yet so I don't fully get it and will probably not use it.

I have another question for you but it's not Leap-specific so I will ask by PM.

Thanks again!

[getnamo]

I found what was the problem. It's related to VR mode. https://developer.leapmotion.com/vr

Was it a toggle you forgot to use?

this plugin is incredible and works great, thanks so much for this integration work!

Glad you found it useful !

when i'm retrieving a world-space value for the distal position of the index finger (or any finger segment position for that matter), i'm storing this value, and converting it to screen space. this has been extremely accurate in windowed mode, however in full screen mode, there seems to be an offset when grabbing this value and converting it. any idea if this is related to the plugin code? i'm trying to pinpoint various locations to check. does full screen enable hmd offset compensation even when there isn't any hmd device enabled?

I believe HMD mode automatically adds the leap-rift offset, which is 8cm FVector(8,0,0) rotated by the HMD. This will only be added if you enable HMD mode.

Hi Getnamo,

In this plugin (and your Myo plugin and your Hydra plugin) you make use of a "delegate" concept. However, looking through your code you do not seem to use any of UE4's delegate system. Did you look at that and, if so, why did you choose your own custom delegate implementation (mostly through subclassing and virtual functions)?

I ask because I'm making my own plugin now and I wonder if I should use UE4's system.

Back when I started with the hydra plugin (4.0) there wasn't as much documentation so I wasn't aware of the UE4 delegate system. I was looking for an architecture that would allow me to add event driven, auto-ticking to any class where you would've wanted it in C++. As time went on and I learned more about UE4 it became clear that components offer more flexibility and you could receive events by adding an interface. This allowed each plugin to now extend functionality directly in blueprints rather than C++. I still believe this, along with including convenience characters/actors that can be used for common use cases (e.g. rigged character), is the best approach.

If you can see a way to make your code cleaner by using the UE delegate system, use it. Let us know how it goes!

[paradoc]

when i'm retrieving a world-space value for the distal position of the index finger (or any finger segment position for that matter), i'm storing this value, and converting it to screen space. this has been extremely accurate in windowed mode, however in full screen mode, there seems to be an offset when grabbing this value and converting it. any idea if this is related to the plugin code? i'm trying to pinpoint various locations to check. does full screen enable hmd offset compensation even when there isn't any hmd device enabled?

[bullale]

Hi Getnamo,

In this plugin (and your Myo plugin and your Hydra plugin) you make use of a "delegate" concept. However, looking through your code you do not seem to use any of UE4's delegate system. Did you look at that and, if so, why did you choose your own custom delegate implementation (mostly through subclassing and virtual functions)?

I ask because I'm making my own plugin now and I wonder if I should use UE4's system.