[Plugin] Leap Motion - Event Driven

[knack]

today my leap upgrade without have checked automatic upgrades.

but works. I read something about API changes on images only (i read fast), check the changelog.

[Fenra]

Knack, good call I did not notice that there was a big change from 2.2.0 to 2.2.1. So, LEAP SDK 2.2.1 is not supported? Uh oh, well I don't have 2.2.0 or lower. I just tried google trying to find the previous version but no luck. I doubt I could edit my version of the leap plugin for the new bindings without messing up somewhere, is there somewhere that I can find leap sdk 2.2.0?

Version 2.2.1

Added APIs for all language bindings (except JavaScript):
Controller.images()
Listener.onImages()
Image.sequenceID()
Added LeapFrame serialize and deserialize functions for Objective-C language binding.

[knack]

i read the latest leap motion sdk have API changes. beware at update. I not update.

[Fenra]

First off, great work on the plug in, this very well done . Unfortunately, I'm running into some problems. My end goal is to use a tool (e.g. pencil) to move a crosshair around. Step 1 is to get a non-tick event to work (like detect hands, etc.), and the images below are my attempt to use the raw image event. The "Hello" statement fires, but "It works?" does not.



Also, I tried to get a tool detected using tick and then used tool->count->print string(count) and it always says the count is zero. I've also tried getting the direction of the tool, but it always at 0,0,0. Image below is one of my many attempts to find a working solution (tick is not connected in the image below). Any suggestions?

[getnamo]

I have a student working on writing a plugin for some of our scientific hardware and we are using your plugins as examples (in addition to actually using the Leap too). We're still learning programming for UE4, but I think a minimal plugin example that actually did something would be tremendously helpful, and much better than what's currently in the Wiki. For maximum community benefit this minimal plugin should probably not require hardware, but then it won't help us debug the OS X problem . I'm not at my office with my toys/research equipment every day but I'll take another look at getting a minimal Leap plugin working sometime next week.

There are two fully barebone plugin code examples that are included in each engine release. If you wan't to see a pretty barebone input bind plugin use the https://github.com/Ikarus76/UEJoystickPlugin as it only binds the windows direct input api to input mapping. The leap plugin is unfortunately to big in my opinion to properly encapsulate in the (FModulePlugin.cpp) module and I chose to set the classes directly to their UObject analogs instead, which works great because the leap SDK offloads that to another thread underneath.

It was a while ago. I guess I didn't remember it correctly. I had assumed it would be possible to code in some file-copy operations into PluginName.build.cs, but upon searching I realized the poster was running a custom script. See jyakku's response to the accepted answer here.

This is what the plugins currently already use to link to libs and dlls, it doesn't copy anything in the packaging process unfortunately. If you link to a lib it expects the .dll to be in the project or engine binaries folder not the plugin one, the only way around it so far is to do a direct dll bind which is not recommended.

Hi,
I have dragged and dropped the binaries, plugins and content to my project root, then restarted UE4. When I launch the project I get this error message below. At this point I did not change any project/game settings, and have tried tested it just with the stater content but I get the same error. Disabling the plugin stops the error and the project runs fine.
...
I would be greatfull for any help.
Thanks, Lewis

Launch is currently not supported, use Play In Editor and Play Standalone for testing. You can also package and ship following the shipping instructions on the github repo.

The reason for this is that currently there is no way to hook into the packaging process to move the required files to the right directory for the project to see the dependencies automatically. Waiting on epic to add necessary fixes before launch will work properly.

[lewisq]

Hi,
I have dragged and dropped the binaries, plugins and content to my project root, then restarted UE4. When I launch the project I get this error message below. At this point I did not change any project/game settings, and have tried tested it just with the stater content but I get the same error. Disabling the plugin stops the error and the project runs fine.

I would be greatfull for any help.
Thanks, Lewis

[knack]

now i checking things about collision and taking objects (my head its overloading after try understand how works all this, rotations, etc and make it work ) but later next week i will try fix that. If i got something i write something.

[bullale]

Good investigation! Haven't had the opportunity to bind anything except windows so far, but I will be traveling with my old mac in the coming weeks so should have an opportunity to see how OSX integration would look like. Will take what you've found and poke at things to see what works and doesn't, I imagine the devil is in the details. We should try something much simpler for a start to see if a basic bind can be achieved and then move from there.

I have a student working on writing a plugin for some of our scientific hardware and we are using your plugins as examples (in addition to actually using the Leap too). We're still learning programming for UE4, but I think a minimal plugin example that actually did something would be tremendously helpful, and much better than what's currently in the Wiki. For maximum community benefit this minimal plugin should probably not require hardware, but then it won't help us debug the OS X problem . I'm not at my office with my toys/research equipment every day but I'll take another look at getting a minimal Leap plugin working sometime next week.

Let me know if you can find a link to the PluginName.build.cs copy modification, this would be quite useful for all plugins with dependencies.

It was a while ago. I guess I didn't remember it correctly. I had assumed it would be possible to code in some file-copy operations into PluginName.build.cs, but upon searching I realized the poster was running a custom script. See jyakku's response to the accepted answer here.

[getnamo]

I supposed that its the finger problem:
https://www.youtube.com/watch?v=Ya_s3WLLAJI
I found too a problem when move the HMD (last minute in video i think) the arms move. Your example proyect have same problem.

Nice video, it does seems like the same problem, let me know if you find a fix. Unsure of the exact cause, but maybe it has to do with how yaw is set by the leap or the finger FK order; to be determined...

...
thanks for the plugin work. Still looking step by step how works i not understand few thinks of that blueprints. Ins't basic your example! :lol:

You're welcome and yeah the rigged character is a little bit complicated because I'm preparing for the transition to input hardware agnostic rigging. For this reason there is a intermediate data structure (anim body) which holds and translates all the data into component space. This allows you to use the same anim body, but set by another hardware (e.g. hydra) and the rigging would remain valid.

Note that thanks to the UE animation retargeting system you can copy animation blueprints (e.g. the basic one) to your own skeletal mesh if you've set up the skeleton bone remapping using the retarget manager.

In OSX, I was able to get the plugin to show up in the list of plugins, but nothing works. Maybe telling is that, even after building, Plugins\LeapMotion\Binaries\Mac is empty.

First, to get it to build for OSX:
+Binaries\Mac\libLeap.dylib (copied from SDK)
+Plugins\LeapMotion\ThirdParty\LeapSDK\Lib\Mac\libLeap.dylib (copied from SDK)
modify Plugins\LeapMotion\Source\LeapMotion\LeapMotion.build.cs to load Mac\libLeap.dylib
The project will then build, and in the editor Window->Plugins, under Installed>Input>Controller, there is Leap Motion Plugin v 0.8.2 and it is checked as enbaled.

However, I can't seem to access it ...

(1) ... using the provided content.
Change the game mode to RenderToTexture.
Change the Default Pawn Class to LeapRiggedCharacter.
Try to launch in viewport, then get a warning:

Clicking on yes gives me "BLUEPRINT COMPILER ERROR" in read text within the viewport.
If I try to edit the LeapRiggedCharacter by double clicking on the blueprint in the content browser, then the editor crashes. This is the message I get in XCode:

[2014.12.17-17.56.00:057][475]LogRHI:Warning: OpenGL debug output extension not supported!
[2014.12.17-17.56.05:856][672]LogRHI:Warning: OpenGL debug output extension not supported!
[2014.12.17-17.56.06:059][672]LogRHI:Warning: OpenGL debug output extension not supported!
Assertion failed: InSCSNode->ComponentTemplate != NULL

(2) ... within a Character blueprint.
I reverted back to the default Game Mode. Made a new Character Class blueprint. Click on Blueprint Props, scroll down to Interfaces, click on the Implemented Interfaces>Add drop down box and Leap is not there.

(3) ... within the Engine Input settings. No Leap inputs can be found in any gamepad, keyboard, mouse, etc.

Any ideas?

By the way, I once read somewhere that you can modify PluginName.build.cs so that it copies the library into MyProject\Binaries\Platform. That way you don't need to provide the Binaries directory in your repository, then you'd be able to refactor your repository so it could simply be copied into the Plugins directory and maybe even be a submodule of another project. (Except for the optional content directory, but I think plugins can/will support content inclusion).

Good investigation! Haven't had the opportunity to bind anything except windows so far, but I will be traveling with my old mac in the coming weeks so should have an opportunity to see how OSX integration would look like. Will take what you've found and poke at things to see what works and doesn't, I imagine the devil is in the details. We should try something much simpler for a start to see if a basic bind can be achieved and then move from there.

Let me know if you can find a link to the PluginName.build.cs copy modification, this would be quite useful for all plugins with dependencies.

[bullale]

OSX support

In OSX, I was able to get the plugin to show up in the list of plugins, but nothing works. Maybe telling is that, even after building, Plugins\LeapMotion\Binaries\Mac is empty.

First, to get it to build for OSX:
+Binaries\Mac\libLeap.dylib (copied from SDK)
+Plugins\LeapMotion\ThirdParty\LeapSDK\Lib\Mac\libLeap.dylib (copied from SDK)
modify Plugins\LeapMotion\Source\LeapMotion\LeapMotion.build.cs to load Mac\libLeap.dylib
The project will then build, and in the editor Window->Plugins, under Installed>Input>Controller, there is Leap Motion Plugin v 0.8.2 and it is checked as enbaled.

However, I can't seem to access it ...

(1) ... using the provided content.
Change the game mode to RenderToTexture.
Change the Default Pawn Class to LeapRiggedCharacter.
Try to launch in viewport, then get a warning:

One or more blueprints has an unresolved compiler error, are you sure you want to Play in Editor?
LeapDebugHand
LeapDebugCharacter
LeapBasicRiggedCharacter
BasicCharacter_AnimBlueprint

Clicking on yes gives me "BLUEPRINT COMPILER ERROR" in read text within the viewport.
If I try to edit the LeapRiggedCharacter by double clicking on the blueprint in the content browser, then the editor crashes. This is the message I get in XCode:

[2014.12.17-17.56.00:057][475]LogRHI:Warning: OpenGL debug output extension not supported!
[2014.12.17-17.56.05:856][672]LogRHI:Warning: OpenGL debug output extension not supported!
[2014.12.17-17.56.06:059][672]LogRHI:Warning: OpenGL debug output extension not supported!
Assertion failed: InSCSNode->ComponentTemplate != NULL

(2) ... within a Character blueprint.
I reverted back to the default Game Mode. Made a new Character Class blueprint. Click on Blueprint Props, scroll down to Interfaces, click on the Implemented Interfaces>Add drop down box and Leap is not there.

(3) ... within the Engine Input settings. No Leap inputs can be found in any gamepad, keyboard, mouse, etc.

Any ideas?

By the way, I once read somewhere that you can modify PluginName.build.cs so that it copies the library into MyProject\Binaries\Platform. That way you don't need to provide the Binaries directory in your repository, then you'd be able to refactor your repository so it could simply be copied into the Plugins directory and maybe even be a submodule of another project. (Except for the optional content directory, but I think plugins can/will support content inclusion).

[knack]

I supposed that its the finger problem:
https://www.youtube.com/watch?v=Ya_s3WLLAJI
I found too a problem when move the HMD (last minute in video i think) the arms move. Your example proyect have same problem.

[knack]

WTF, after hours and hours, and near do a harakiri ritual... i found what was failing.

My anigraph was something like this...: (i not remember exact)

LeftElbowTransform->RightHandTransform->RightElbowTransform->LeftHandTransform


i change to the order to your template order and fix the issue... dammit!



thanks for the plugin work. Still looking step by step how works i not understand few thinks of that blueprints. Ins't basic your example! :lol:

[getnamo]

i have a problem with rotation of the hands of my custom character. its rigged with ART.

Anyone known where are the problem? in the video i try clapping and show my palm in front of my head.

Here the video:
https://www.youtube.com/watch?v=1VXQdHgom6g

The code its copy the necessary nodes from plugin content, but not all, perhaps i missed something?

It looks like your hands are rotating around YAW whenever they cross some midpoint line. I encountered something similar with the fingers in the rigged model and didn't have the time to correct it, but the wrist rotation doesn't have this problem. It important to remember that you need to combine rotations when you change orientation in blueprint and to avoid using break rotation/make rotation unless you know what you're doing. If you do not heed that, you may get gimbal locks.

The general format of the rigging looks like this: Leap Positions and Orientations in UE space (X forward), Leap basis (Leap Debug Hand location) subtracted from component (converts leap position to component space) -> Rotated by 90 degrees to have Y forward for animation space. Use these rotations/positions in your rigging.

[knack]

... duplicate ...

[knack]

i have a problem with rotation of the hands of my custom character. its rigged with ART.

Anyone known where are the problem? in the video i try clapping and show my palm in front of my head.

Here the video:
https://www.youtube.com/watch?v=1VXQdHgom6g

The code its copy the necessary nodes from plugin content, but not all, perhaps i missed something?