[Plugin] Leap Motion - Event Driven

[vimaxus]

This is pretty cool! I noticed you made a pull request, I will take a look and see how to best integrate this. What is it that you are tracking in this example?

Well, I was tracking just some blobs in the image. If you mean in real-life, as I couldn't find what zalo suggested (motion tracking spheres) I covered some polystyrene balls (3.5cm) with 3M reflecting tape to get sort of equivalent.
I tried with small pathes initially (1.5cm squares) on my mobile phone but if I went over 30-40cm away from the leap I got very inconsistent results. The resolution is very low on the leap, well, actually the resolution coupled with the huge angle gives a too greater solid angle for each pixel so you need BIG things to track.

Here is how I used it at the time:


As I said in the note, I find the CPU use exaggerated, so I'm trying to export 3 vectors for each blob instead of one, the other 2 being coordinates on the image, in order to be able to use some condition so that when I satisfy a list of conditions of relative distances from each other. (like a triange, or a sqare with diagonals) I "lock" only on the blobs and only search in the vicinity of where they were the last frame. It's looking great, it's a lot better CPU wise, but now I'm trying to change the recursive flood with just a regular pass on a square surrounding the last position.

Also, if anyone knows some algorithms to eliminate a high-frequency shake from my measurements (but not averaging as not to introduce latency) I'm very interested. I remember reading something of the sort on an Oculus blob post, quite a while back.

edit: I initially worked on a version in LeapImageList so as to do directly the 2 images and return directly the 3d blobs, to avoid the 2 separate branches in blueprint, but while it worked I got often crashes (sometimes around the 60sec mark, sometimes later) relating to setLeapImage. Maybe I was doing something wrong but I basically kept the 2 pointers that you used and re-defined them at each call. If you want I can send the related files, but I eventually thought that maybe, someone would like the separate results by image...

[getnamo]

I managed to get the blob tracking working. It's really just the same as Zalo's script so all credit goes to him and of course getnamo. I'll try to send a pm with it to getnamo hopefully to be included in his plugin. (not sure how github branching works and I prefer to avoid it if I can)

https://www.youtube.com/watch?v=QYFN...ature=youtu.be

you can see the red/blue spheres as debug spheres for blob positions in images and green spheres with 3d positions of blobs (the other fixed ones are the halogen lamps)

This is pretty cool! I noticed you made a pull request, I will take a look and see how to best integrate this. What is it that you are tracking in this example?

This plugin is not supported for 4.6th version

And

In 4.5th version Plugin is supporting well, but while exporting to .EXE for windows 64 and 32 bit it rises error, its not working....

I'm using windows 8 64 bit

...

I may need more information, describe in more detail what you are trying to do. When you say exporting to .exe, do you mean packaging for shipping/development? Please make sure you follow the shipping instructions found at the github repo if that is the case. If you are still having problems, PM me your logs found in {Project Root}/Saved/Logs

[a.carter1182]

Works fine on my 4.6 install with getnamo's latest + leap 2.2.......I've never tried to export yet so I cant comment there.

[Rasa]

This plugin is not supported for 4.6th version

And

In 4.5th version Plugin is supporting well, but while exporting to .EXE for windows 64 and 32 bit it rises error, its not working....

I'm using windows 8 64 bit

[a.carter1182]

Heh......thought that was a sarcastic remark at first........but that seems to be a legit software/hardware that I really might be able to make work for this application. Great job on the code work and I'll be able to push further as this comes along. Also thanks getnamo as the collision now works after sockets with some cubes attached.

[vimaxus]

Thats great vimaxus!! Im trying to think in my head how I can break a mouse apart and use the LMB + guts of mouse and make either a wooden gun or an attachment to our simulated firearms in order to trigger the firing after I've setup the trackers on the item. So for his script did you just have to essentially change things like mathf to fmath, etc for it to work in C++ ?

http://www.makeymakey.com/ ?

it's not that straightforward, you'll have access to the source soon enough. I'm also working on a blueprint to lock on a certain blob (or you could just sort the blobs) because for example the 3rd one from the vector array isn't necessarily the same blob all the time.

[a.carter1182]

Thats great vimaxus!! Im trying to think in my head how I can break a mouse apart and use the LMB + guts of mouse and make either a wooden gun or an attachment to our simulated firearms in order to trigger the firing after I've setup the trackers on the item. So for his script did you just have to essentially change things like mathf to fmath, etc for it to work in C++ ?

[vimaxus]

I managed to get the blob tracking working. It's really just the same as Zalo's script so all credit goes to him and of course getnamo. I'll try to send a pm with it to getnamo hopefully to be included in his plugin. (not sure how github branching works and I prefer to avoid it if I can)

https://www.youtube.com/watch?v=QYFN...ature=youtu.be

you can see the red/blue spheres as debug spheres for blob positions in images and green spheres with 3d positions of blobs (the other fixed ones are the halogen lamps)

[a.carter1182]

Ahhh ok, essentially taking it back to capsule collider that is like the default on a character with the camera, etc. I got you thanks for that...didnt think to do that. I figured it would be a long shot on the ps move, and that marker tracking looks useable I really appreciate that link. As always thanks for your great work.

[getnamo]

Maybe Im going about this the wrong way.....but I cant figure out how to make collisions happen with the hands/fingers. The body was already colliding but no setups/checkboxes etc. have seemed to work. Could you provide me with a bit of knowledge please sir.........also quick question, more towards your work with hydra plugin.....but I and most likely others.....havent been able to purchase a hydra system. No telling when STEM is coming out, but I know at some point somebody was using your work I think to attempt the same result with the PS Move. He/She got close but I believe it was some camera drivers or something that stopped it. Anyhow wondering if a PS Move plugin might be a project you might startup one day. Would be nice as I have no solution for shooting in VR that is not immersion breaking yet.....as in I dont want a xbox controller......at least some sort of gun like device the Move does and not really sure if the Hydra has one. Either way great work sir I wish I could get my programming skills up enough that I could contribute to the VR community.

There are a couple ways to get collision to work. Probably the easiest one would be to add box/capsule components which are parented to sockets for fingers/ hand positions. Refer to SkeletalMesh Socket documentation and Collision documentation for details.

I am unlikely to bind the PS Move but if you want to take a crack at it, remember that UE4 requires 64bit dlls in order to use in the editor, which I believe was the issue with the previous attempt by n00854180t. Refer to any of my plugins for reference on how to achieve a plugin bind, but without a 64bit dll you're unlikely to be successful. Best bet is to use devices which are out now like the Leap and Myo or wait for future VR devices. At least one user (Zalo) has been working on tracking other objects through the leap see retroreflective marker forum post for details.


Update to 0.8.2
-Now supports UE 4.6

[a.carter1182]

Maybe Im going about this the wrong way.....but I cant figure out how to make collisions happen with the hands/fingers. The body was already colliding but no setups/checkboxes etc. have seemed to work. Could you provide me with a bit of knowledge please sir.........also quick question, more towards your work with hydra plugin.....but I and most likely others.....havent been able to purchase a hydra system. No telling when STEM is coming out, but I know at some point somebody was using your work I think to attempt the same result with the PS Move. He/She got close but I believe it was some camera drivers or something that stopped it. Anyhow wondering if a PS Move plugin might be a project you might startup one day. Would be nice as I have no solution for shooting in VR that is not immersion breaking yet.....as in I dont want a xbox controller......at least some sort of gun like device the Move does and not really sure if the Hydra has one. Either way great work sir I wish I could get my programming skills up enough that I could contribute to the VR community.

[Xeno]

You'r Fantasticcccccccc
Thank youu *

[getnamo]

One thing off of the top of my head: when you added a leap component to your blueprint, did you use the direct method or event graph method? The reason I ask is that sometimes people make the mistake of doing both, which will duplicate the component, its supposed to be either one or the other, never both. Your log will output a warning any time you have more than one leap component registered (note that while in the editor, it may report a false positive due to the editor adding an extra component during component preview).

If that doesn't solve it, pm me some of your recent logs found in Saved/Logs, any logs which resulted in a crash caused by this would be most helpful in finding the root cause.

With the logs sent from Mega Future Games I was able to identify the memory leaks that were causing this and I pushed an update with these fixes, as well as other API that was missing from LeapFrame.

Update to 0.8.1
-Fixed Memory leaks which would cause crash around 60second mark (GC
interval) for list-based objects using extended and append
-Added appendTools and appendFingers to pointableList
-LeapFrame got its API updated to include all of the missing functions,
should now reflect the leap api nearly completely (except 64bit values
in blueprint that are not supported)
-Added log entry with plugin version at game start for easier debugging

Also added some simple setup gifs to show how easy it is to use the basic rigged setup:
Setting your character


Play Test

[getnamo]

Thanks for getting back to me so quick.

I did as you suggested and deleted the plugin completely and re-added the newer version. Now the engine sees it as ver. 0.8. I reimplemented my code with the updated functions from the new version.

It hasn't seemed to help unfortunately. Ive made a short screen cast showing the problem: https://www.youtube.com/watch?v=g7pu...ature=youtu.be
In the video, the circle gesture event fires repeatedly (each time it changes the colour of the block on the table, has a 1.5 second cool down, then fires again). Once the time hits one minute, it stops firing. Although you cant see my hand, it still preforms the circle gesture after the minute mark.

I assume the same root cause is whats causing the polling gesture to crash as well so I've not yet tested that with the update (that's next on the to do list while I wait a reply). Im sure its something simple I'm just not noticing.

Edit: I tested the Polled Gesture and it did go well either. It goes past the minute marker and still regression the gesture though.The game exits fine with the code as is, if you don't preform the gesture. If you do preform it at least once, the game carries on fine but when you go to exit the game it crashes. It gives different reasons for doing so in the crash report each time.

Still, just letting everyone know. One problem at a time, I guess.

One thing off of the top of my head: when you added a leap component to your blueprint, did you use the direct method or event graph method? The reason I ask is that sometimes people make the mistake of doing both, which will duplicate the component, its supposed to be either one or the other, never both. Your log will output a warning any time you have more than one leap component registered (note that while in the editor, it may report a false positive due to the editor adding an extra component during component preview).

If that doesn't solve it, pm me some of your recent logs found in Saved/Logs, any logs which resulted in a crash caused by this would be most helpful in finding the root cause.

[Mega Future Games]

Thanks for getting back to me so quick.

I did as you suggested and deleted the plugin completely and re-added the newer version. Now the engine sees it as ver. 0.8. I reimplemented my code with the updated functions from the new version.

It hasn't seemed to help unfortunately. Ive made a short screen cast showing the problem: https://www.youtube.com/watch?v=g7pu...ature=youtu.be
In the video, the circle gesture event fires repeatedly (each time it changes the colour of the block on the table, has a 1.5 second cool down, then fires again). Once the time hits one minute, it stops firing. Although you cant see my hand, it still preforms the circle gesture after the minute mark.

I assume the same root cause is whats causing the polling gesture to crash as well so I've not yet tested that with the update (that's next on the to do list while I wait a reply). Im sure its something simple I'm just not noticing.

Edit: I tested the Polled Gesture and it did go well either. It goes past the minute marker and still regression the gesture though.The game exits fine with the code as is, if you don't preform the gesture. If you do preform it at least once, the game carries on fine but when you go to exit the game it crashes. It gives different reasons for doing so in the crash report each time.

Still, just letting everyone know. One problem at a time, I guess.