[Plugin] Leap Motion - Event Driven

[vhung92]

This might be a load order problem, see if changing the load order inside https://github.com/getnamo/leap-ue4/...Motion.uplugin would solve that (try predefault).

Hey getnamo, thanks for the reply! I tried adding the line

"LoadingPhase": "PreDefault"

but I still have the same issue. The workaround is not really hindering me in any way and I would like to ask another question regarding the Leap and UMG.

I would like to have Leap interaction with a 3D menu that I have. I've managed to attach the menu to my custom leap character (subclassing LeapCollisionCharacter) because I want the menu to follow the camera movements of the user as shown below:


However, I'm not entirely sure where to start in order to get the menu to interact with the hand. I found this link https://answers.unrealengine.com/que...or-3d-umg.html where you said you are using the same hack but I'm guessing instead of using the mouse position you use the hand position translated to (x,y) that you described here. Is it possible for you to put up the blueprint you were using or a tutorial on how to dit?

EDIT: I've tried first by adding OnComponentBeginOverlap(menu) as well as adding a box collider and using OnComponentBeginOverlap(box) to just print the location of the hit (breaking the hit result) but it's not generating any overlap events.


EDIT2: So I figured out what was wrong, I had forgotten to enable "Generate Overlap Events" on the hand meshes and I can now get events when the hand overlaps with the menu widget, Now I'm looking at translating the 3d world location of the hand to the widgets local space as well as calling functions on the menu widget.


Thanks in advance!

[Ders92]

I think that the right place should still be https://github.com/getnamo/leap-ue4/...aceUtility.cpp. E.g. multiplying the UE_TO_LEAP_SCALE by w2m inside the code, will try this approach next time I do a plugin pass.

Everything works now! This is what solves it in the function convertAndScaleLeapToUE:
float w2m = GWorld->GetWorldSettings()->AWorldSettings::WorldToMeters / 100.0f;
FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * w2m, leapVector.x * LEAP_TO_UE_SCALE * w2m, leapVector.y * LEAP_TO_UE_SCALE * w2m);

this is something i tried at first but the thing i missed was that the size of the hand meshes has to be scaled with the same factor (w2m). So I just exported your hands to blender, scaled them and then reimported them and force replaced the old hands_lo and its skeleton with the new. I got a weird orientation error but it was solved by checking the box "TOas ref pose" under mesh settings when importing.

[Rukasu]

Hey,

I've encountered some problem using leapmotion + player teleportation/rotation.

I am working on a project that uses the oculus, to get a view of the backside we teleport the player to the new location and rotate him 180 degrees, so he looks straigth at the backside of the object.
The problem is, the controller (LeapEchoCollusionCharacter) doesn't rotate with the player. That means the right hand always shows up, but the left hand doesn't. It is positioned behind the player and you have to look at it first in order to be able to use it.

I need a workaround for this, 'cause we can't have it this way.

[Fnordcorps]

Hi Getnamo, thanks for your plugin, we have had it working brilliantly. I was wondering if you are planning to update to 4.11 anytime soon? Unfortunately - trying and failing to be clever and 'future proof' we decided to build our project in 4.11 and now we want to integrate leap motion UE4 throws up a load of errors when we try to put the plugin in our project. Screenshots of the errors attached if they are of any help to you. thanks again!

[Mrob76u]

Hey getnamo

A completely crazy idea. would it be possible to get the leap motion hands working in the VR editor instead of the controllers with gestures as the buttons. See, told you it was a crazy idea

[getnamo]

I just put my leap experiment up on github if you want to have a look. https://github.com/opamp77/UnrealEng....10Experiments

Unfortunalty after thurther testing this does'nt appear to totally remove the rotational latency only reduce it but maybe im doing something wrong.

As far as the Orion branch is concerned I noticed a few merges into the Master/4.11 branch after looking at a few more of them It appears that Orion is Epics internal name for Paragon.

Looking ahead it looks like an applylateupdate() function is now being called within the hmd plugins.

https://github.com/EpicGames/UnrealE...pplyLateUpdate

It looks like in 4.11 you can inform the HeadmountedDisplay module of components that you would like updating on the render thread.

https://github.com/EpicGames/UnrealE...etuplateupdate

And it look like the camera gets a late update by default.

If im right this should make things a bit easier.

This is really cool. I've had some success with timewarp but it jitters like mad (I think overdriving is the only way to have it work, but it introduces the jitter). The most reliable way of getting tracking performance seems to be to re-factor the plugin for input device module with late update and attaching it to a camera component. Full timewarp though is still eluding me. I'll share my version once it's a bit cleaner.

Okey thanks for answering! One thing I want to add is that the built in plugin uses the "world to meters" parameter in their cpp file that I believe corresponds to your LeapHand.cpp file, the location of this class is here in the unreal engine source code: UnrealEngine\Engine\Plugins\Runtime\LeapMotionController\Source\LeapMotionController\Private\LeapMotionHandActor.cpp

So they seem to take into account w2m in their code that updates the position. i tried doing something similiar in your plugin but as I wrote in my thread it had no success. But that might be because i dont fully understand how everything is put together :P However I have contacted one of the epic staff about this and hope that someone that made the built in plugin can shed some light on this issue.

I think that the right place should still be https://github.com/getnamo/leap-ue4/...aceUtility.cpp. E.g. multiplying the UE_TO_LEAP_SCALE by w2m inside the code, will try this approach next time I do a plugin pass.

Hello,

I'm currently using this plugin to try and create an editor for manipulating virtual objects and I'm having some trouble with the plugin and source control (git). I have an existing project that works when I clone it from github and it works after adding the Pugins/ and Binaries/ folders, but any commit I make after adding these folders just breaks the blueprints (even if it is just a single line of c++ code). For example I set my DefaultPawnClass in my custom game mode in code using:



and by adding this I get the following errors when I shut down the editor and start it again (through VS2015, DebugGame Editor):
...

Any idea what causes this or how to solve it?

EDIT:
It seems like it has nothing to do with git whatsoever, but rather the above code snippet. If I set the default pawn, whether it be c++ or through a GameMode BP, it results in a load error IF I restart the editor. The current workaround is to start the editor with another default pawn and when the editor is running, change it back to LeapCollisionCharacter (or any subclass of it).

This might be a load order problem, see if changing the load order inside https://github.com/getnamo/leap-ue4/...Motion.uplugin would solve that (try predefault).

Hello. We have been using your plugin, and it has been working great. However, we need to be able to scale the hands and move them further away from us (so that we are bigger), but after hours of trying we can not figure out how to do it. Is there a way to do this?

You can use the blueprint custom approach and do whatever you need with the data, or you can modify e.g. convertAndScaleLeapToUE in https://github.com/getnamo/leap-ue4/...aceUtility.cpp to modify the scale. Probably adding support for world2meter and custom scaling should be added to the plugin. If you make a modification that works, consider making a pull request in the plugin.

Hey. I would like to change the skeletal mesh from the yellow guy to a custom one. Is there any tutorials or docs on how to accomplish this? Been googling for hours without any result.

I'm still working on body-state which should make this easier in the future but for now the best approach is to use Animation Retargeting.

[novalain]

Hey. I would like to change the skeletal mesh from the yellow guy to a custom one. Is there any tutorials or docs on how to accomplish this? Been googling for hours without any result.

[xNoxerx]

Hello. We have been using your plugin, and it has been working great. However, we need to be able to scale the hands and move them further away from us (so that we are bigger), but after hours of trying we can not figure out how to do it. Is there a way to do this?

[vhung92]

Hello,

I'm currently using this plugin to try and create an editor for manipulating virtual objects and I'm having some trouble with the plugin and source control (git). I have an existing project that works when I clone it from github and it works after adding the Pugins/ and Binaries/ folders, but any commit I make after adding these folders just breaks the blueprints (even if it is just a single line of c++ code). For example I set my DefaultPawnClass in my custom game mode in code using:

static ConstructorHelpers::FClassFinder<APawn> PlayerPawnClassFinder(TEXT("/LeapMotion/Blueprints/LeapCollisionCharacter"));

and by adding this I get the following errors when I shut down the editor and start it again (through VS2015, DebugGame Editor):


Any idea what causes this or how to solve it?

EDIT:
It seems like it has nothing to do with git whatsoever, but rather the above code snippet. If I set the default pawn, whether it be c++ or through a GameMode BP, it results in a load error IF I restart the editor. The current workaround is to start the editor with another default pawn and when the editor is running, change it back to LeapCollisionCharacter (or any subclass of it).

[Ders92]

Still no better on this issue, I'm hoping someone else has a better understanding of what happens when you change scale. The thread for this problem is here: https://forums.unrealengine.com/show...ameter-problem

Okey thanks for answering! One thing I want to add is that the built in plugin uses the "world to meters" parameter in their cpp file that I believe corresponds to your LeapHand.cpp file, the location of this class is here in the unreal engine source code: UnrealEngine\Engine\Plugins\Runtime\LeapMotionController\Source\LeapMotionController\Private\LeapMotionHandActor.cpp

So they seem to take into account w2m in their code that updates the position. i tried doing something similiar in your plugin but as I wrote in my thread it had no success. But that might be because i dont fully understand how everything is put together :P However I have contacted one of the epic staff about this and hope that someone that made the built in plugin can shed some light on this issue.

[Opamp77]

That's an interesting approach, I'll have to look into some of the bind points you suggested. I know Motion Controllers get a late update function call, but I don't know if regular InputModules do too. There could be a way for the leap motion to pretend it's a motion controller module to get this late update without engine fix. That could be a fairly sizable refactor.

Regarding the branch, how do you find these branches? I haven't found a good way to browser UE branches on github. Sounds fascinating either way... not that I would know anything about that

I just put my leap experiment up on github if you want to have a look. https://github.com/opamp77/UnrealEng....10Experiments

Unfortunalty after thurther testing this does'nt appear to totally remove the rotational latency only reduce it but maybe im doing something wrong.

As far as the Orion branch is concerned I noticed a few merges into the Master/4.11 branch after looking at a few more of them It appears that Orion is Epics internal name for Paragon.

Looking ahead it looks like an applylateupdate() function is now being called within the hmd plugins.

https://github.com/EpicGames/UnrealE...pplyLateUpdate

It looks like in 4.11 you can inform the HeadmountedDisplay module of components that you would like updating on the render thread.

https://github.com/EpicGames/UnrealE...etuplateupdate

And it look like the camera gets a late update by default.

If im right this should make things a bit easier.

[getnamo]

Hello,

somehow I do not manage to see the leap motion image in my scene.
I have added a leap controller in my level BP and did enableImageSupport.
...
I have a cube in my scene with a billboard compontent. The sprite is scaled to screen size with factor 0.25.
...
I have added the key 1 event BP in order to see the leap motion image, but I actually is not showing up. Do you have any hints, what I might do wrong?

Thanks

RalfR

Make sure your sprite isn't set to hidden in game. While your bps look correct you can also try the EventRawImageReceived to get images.

Again thank you for your answer it partly solved my issue as I wrote earlier. However the leap hands seems to get updated weirdly and not adapting to the change in "world to meters". So I looked through the source code for your plugin and found a function you use called "convertAndScaleLeapToUE" in LeapInterfaceUtility.cpp that is used when translating the hands. So my thought is that perhaps the worldToMeters variable should be used in this function so the hands can adapt to the change in scale.

I tried to change the leap hands translation like this:
FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * worldToMeters/100.0,
leapVector.x * LEAP_TO_UE_SCALE *worldToMeters/100.0 ,
leapVector.y * LEAP_TO_UE_SCALE * worldToMeters/100.0);

But this has no effect (tried with extreme values) and im suspecting its because I use blueprints and that the c++ files aren't connected to the blueprints? So my question is if there is a way to achieve this in blueprints or do I have to use your plugin with c++, or is there another way to fix my issue?

Thanks in advance!

Still no better on this issue, I'm hoping someone else has a better understanding of what happens when you change scale. The thread for this problem is here: https://forums.unrealengine.com/show...ameter-problem

It involves engine mod's unfortunately.
Im using the same render thread delegate that I mentioned in the portal thread.
The delegate is an addition the the Headmounted display module.
A modifcation to the oculus plugin broadcast's to the delegate the final eye pose for each camera when it is usually calculated.

the bone's in this instance are staticmeshcomponents attached to hand actors.
At beginplay the LeapManager actor registers a callback function UpdateHandsOn_RT() with the delegate.

on GameThread Tick() the hand bones and update normally using the latest leap::frame at that point in time along with the results of GetHMDOrientationandPosition().

when the oculus plugin has calculated the final eye pose on the render thread it broadcast's location,rotation and Eye.
This fire's off the previously registed UpdateHandsOn_RT() in leaphandmanager.

This in turn calls ApplyLateUpdateTransform() on the sceneproxy's of each of the hands staticmeshcomponents using the latest availible leap frame along with the left eye pose.

I believe ApplyLateUpdateTransform() was implimented in 4.10 to reduce the lag for motioncontrollers.

Until we have a way of accessing the HMD's final pose on the render thread we are reduced to hacking the engine.
Unless there a mechanism allready in place that im not aware of.
But like I said Epic seem to have a branch named orion!?!

I'll put the experiment up on github if you want to have a look?
The reduced latency definately helps with hand prescence.

That's an interesting approach, I'll have to look into some of the bind points you suggested. I know Motion Controllers get a late update function call, but I don't know if regular InputModules do too. There could be a way for the leap motion to pretend it's a motion controller module to get this late update without engine fix. That could be a fairly sizable refactor.

Regarding the branch, how do you find these branches? I haven't found a good way to browser UE branches on github. Sounds fascinating either way... not that I would know anything about that

[RalfRab]

Hello,

somehow I do not manage to see the leap motion image in my scene.
I have added a leap controller in my level BP and did enableImageSupport.

I have a cube in my scene with a billboard compontent. The sprite is scaled to screen size with factor 0.25.

I have added the key 1 event BP in order to see the leap motion image, but I actually is not showing up. Do you have any hints, what I might do wrong?

Thanks

RalfR

[Ders92]

Not sure about this one, first time I've seen this issue. You can try to correct world to meters using ipd instead (stereo e=<ipd value here>)

Again thank you for your answer it partly solved my issue as I wrote earlier. However the leap hands seems to get updated weirdly and not adapting to the change in "world to meters". So I looked through the source code for your plugin and found a function you use called "convertAndScaleLeapToUE" in LeapInterfaceUtility.cpp that is used when translating the hands. So my thought is that perhaps the worldToMeters variable should be used in this function so the hands can adapt to the change in scale.

I tried to change the leap hands translation like this:
FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * worldToMeters/100.0,
leapVector.x * LEAP_TO_UE_SCALE *worldToMeters/100.0 ,
leapVector.y * LEAP_TO_UE_SCALE * worldToMeters/100.0);

But this has no effect (tried with extreme values) and im suspecting its because I use blueprints and that the c++ files aren't connected to the blueprints? So my question is if there is a way to achieve this in blueprints or do I have to use your plugin with c++, or is there another way to fix my issue?

Thanks in advance!

[vimaxus]

I'll put the experiment up on github if you want to have a look?

Pretty pretty please? I for one would really need that and I can't find the time to try it (I wouldn't get it to work as well either).