[Plugin] Leap Motion - Event Driven

[Mrob76u]

Hey getnamo

A completely crazy idea. would it be possible to get the leap motion hands working in the VR editor instead of the controllers with gestures as the buttons. See, told you it was a crazy idea

[getnamo]

I just put my leap experiment up on github if you want to have a look. https://github.com/opamp77/UnrealEng....10Experiments

Unfortunalty after thurther testing this does'nt appear to totally remove the rotational latency only reduce it but maybe im doing something wrong.

As far as the Orion branch is concerned I noticed a few merges into the Master/4.11 branch after looking at a few more of them It appears that Orion is Epics internal name for Paragon.

Looking ahead it looks like an applylateupdate() function is now being called within the hmd plugins.

https://github.com/EpicGames/UnrealE...pplyLateUpdate

It looks like in 4.11 you can inform the HeadmountedDisplay module of components that you would like updating on the render thread.

https://github.com/EpicGames/UnrealE...etuplateupdate

And it look like the camera gets a late update by default.

If im right this should make things a bit easier.

This is really cool. I've had some success with timewarp but it jitters like mad (I think overdriving is the only way to have it work, but it introduces the jitter). The most reliable way of getting tracking performance seems to be to re-factor the plugin for input device module with late update and attaching it to a camera component. Full timewarp though is still eluding me. I'll share my version once it's a bit cleaner.

Okey thanks for answering! One thing I want to add is that the built in plugin uses the "world to meters" parameter in their cpp file that I believe corresponds to your LeapHand.cpp file, the location of this class is here in the unreal engine source code: UnrealEngine\Engine\Plugins\Runtime\LeapMotionController\Source\LeapMotionController\Private\LeapMotionHandActor.cpp

So they seem to take into account w2m in their code that updates the position. i tried doing something similiar in your plugin but as I wrote in my thread it had no success. But that might be because i dont fully understand how everything is put together :P However I have contacted one of the epic staff about this and hope that someone that made the built in plugin can shed some light on this issue.

I think that the right place should still be https://github.com/getnamo/leap-ue4/...aceUtility.cpp. E.g. multiplying the UE_TO_LEAP_SCALE by w2m inside the code, will try this approach next time I do a plugin pass.

Hello,

I'm currently using this plugin to try and create an editor for manipulating virtual objects and I'm having some trouble with the plugin and source control (git). I have an existing project that works when I clone it from github and it works after adding the Pugins/ and Binaries/ folders, but any commit I make after adding these folders just breaks the blueprints (even if it is just a single line of c++ code). For example I set my DefaultPawnClass in my custom game mode in code using:



and by adding this I get the following errors when I shut down the editor and start it again (through VS2015, DebugGame Editor):
...

Any idea what causes this or how to solve it?

EDIT:
It seems like it has nothing to do with git whatsoever, but rather the above code snippet. If I set the default pawn, whether it be c++ or through a GameMode BP, it results in a load error IF I restart the editor. The current workaround is to start the editor with another default pawn and when the editor is running, change it back to LeapCollisionCharacter (or any subclass of it).

This might be a load order problem, see if changing the load order inside https://github.com/getnamo/leap-ue4/...Motion.uplugin would solve that (try predefault).

Hello. We have been using your plugin, and it has been working great. However, we need to be able to scale the hands and move them further away from us (so that we are bigger), but after hours of trying we can not figure out how to do it. Is there a way to do this?

You can use the blueprint custom approach and do whatever you need with the data, or you can modify e.g. convertAndScaleLeapToUE in https://github.com/getnamo/leap-ue4/...aceUtility.cpp to modify the scale. Probably adding support for world2meter and custom scaling should be added to the plugin. If you make a modification that works, consider making a pull request in the plugin.

Hey. I would like to change the skeletal mesh from the yellow guy to a custom one. Is there any tutorials or docs on how to accomplish this? Been googling for hours without any result.

I'm still working on body-state which should make this easier in the future but for now the best approach is to use Animation Retargeting.

[novalain]

Hey. I would like to change the skeletal mesh from the yellow guy to a custom one. Is there any tutorials or docs on how to accomplish this? Been googling for hours without any result.

[xNoxerx]

Hello. We have been using your plugin, and it has been working great. However, we need to be able to scale the hands and move them further away from us (so that we are bigger), but after hours of trying we can not figure out how to do it. Is there a way to do this?

[vhung92]

Hello,

I'm currently using this plugin to try and create an editor for manipulating virtual objects and I'm having some trouble with the plugin and source control (git). I have an existing project that works when I clone it from github and it works after adding the Pugins/ and Binaries/ folders, but any commit I make after adding these folders just breaks the blueprints (even if it is just a single line of c++ code). For example I set my DefaultPawnClass in my custom game mode in code using:

static ConstructorHelpers::FClassFinder<APawn> PlayerPawnClassFinder(TEXT("/LeapMotion/Blueprints/LeapCollisionCharacter"));

and by adding this I get the following errors when I shut down the editor and start it again (through VS2015, DebugGame Editor):


Any idea what causes this or how to solve it?

EDIT:
It seems like it has nothing to do with git whatsoever, but rather the above code snippet. If I set the default pawn, whether it be c++ or through a GameMode BP, it results in a load error IF I restart the editor. The current workaround is to start the editor with another default pawn and when the editor is running, change it back to LeapCollisionCharacter (or any subclass of it).

[Ders92]

Still no better on this issue, I'm hoping someone else has a better understanding of what happens when you change scale. The thread for this problem is here: https://forums.unrealengine.com/show...ameter-problem

Okey thanks for answering! One thing I want to add is that the built in plugin uses the "world to meters" parameter in their cpp file that I believe corresponds to your LeapHand.cpp file, the location of this class is here in the unreal engine source code: UnrealEngine\Engine\Plugins\Runtime\LeapMotionController\Source\LeapMotionController\Private\LeapMotionHandActor.cpp

So they seem to take into account w2m in their code that updates the position. i tried doing something similiar in your plugin but as I wrote in my thread it had no success. But that might be because i dont fully understand how everything is put together :P However I have contacted one of the epic staff about this and hope that someone that made the built in plugin can shed some light on this issue.

[Opamp77]

That's an interesting approach, I'll have to look into some of the bind points you suggested. I know Motion Controllers get a late update function call, but I don't know if regular InputModules do too. There could be a way for the leap motion to pretend it's a motion controller module to get this late update without engine fix. That could be a fairly sizable refactor.

Regarding the branch, how do you find these branches? I haven't found a good way to browser UE branches on github. Sounds fascinating either way... not that I would know anything about that

I just put my leap experiment up on github if you want to have a look. https://github.com/opamp77/UnrealEng....10Experiments

Unfortunalty after thurther testing this does'nt appear to totally remove the rotational latency only reduce it but maybe im doing something wrong.

As far as the Orion branch is concerned I noticed a few merges into the Master/4.11 branch after looking at a few more of them It appears that Orion is Epics internal name for Paragon.

Looking ahead it looks like an applylateupdate() function is now being called within the hmd plugins.

https://github.com/EpicGames/UnrealE...pplyLateUpdate

It looks like in 4.11 you can inform the HeadmountedDisplay module of components that you would like updating on the render thread.

https://github.com/EpicGames/UnrealE...etuplateupdate

And it look like the camera gets a late update by default.

If im right this should make things a bit easier.

[getnamo]

Hello,

somehow I do not manage to see the leap motion image in my scene.
I have added a leap controller in my level BP and did enableImageSupport.
...
I have a cube in my scene with a billboard compontent. The sprite is scaled to screen size with factor 0.25.
...
I have added the key 1 event BP in order to see the leap motion image, but I actually is not showing up. Do you have any hints, what I might do wrong?

Thanks

RalfR

Make sure your sprite isn't set to hidden in game. While your bps look correct you can also try the EventRawImageReceived to get images.

Again thank you for your answer it partly solved my issue as I wrote earlier. However the leap hands seems to get updated weirdly and not adapting to the change in "world to meters". So I looked through the source code for your plugin and found a function you use called "convertAndScaleLeapToUE" in LeapInterfaceUtility.cpp that is used when translating the hands. So my thought is that perhaps the worldToMeters variable should be used in this function so the hands can adapt to the change in scale.

I tried to change the leap hands translation like this:
FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * worldToMeters/100.0,
leapVector.x * LEAP_TO_UE_SCALE *worldToMeters/100.0 ,
leapVector.y * LEAP_TO_UE_SCALE * worldToMeters/100.0);

But this has no effect (tried with extreme values) and im suspecting its because I use blueprints and that the c++ files aren't connected to the blueprints? So my question is if there is a way to achieve this in blueprints or do I have to use your plugin with c++, or is there another way to fix my issue?

Thanks in advance!

Still no better on this issue, I'm hoping someone else has a better understanding of what happens when you change scale. The thread for this problem is here: https://forums.unrealengine.com/show...ameter-problem

It involves engine mod's unfortunately.
Im using the same render thread delegate that I mentioned in the portal thread.
The delegate is an addition the the Headmounted display module.
A modifcation to the oculus plugin broadcast's to the delegate the final eye pose for each camera when it is usually calculated.

the bone's in this instance are staticmeshcomponents attached to hand actors.
At beginplay the LeapManager actor registers a callback function UpdateHandsOn_RT() with the delegate.

on GameThread Tick() the hand bones and update normally using the latest leap::frame at that point in time along with the results of GetHMDOrientationandPosition().

when the oculus plugin has calculated the final eye pose on the render thread it broadcast's location,rotation and Eye.
This fire's off the previously registed UpdateHandsOn_RT() in leaphandmanager.

This in turn calls ApplyLateUpdateTransform() on the sceneproxy's of each of the hands staticmeshcomponents using the latest availible leap frame along with the left eye pose.

I believe ApplyLateUpdateTransform() was implimented in 4.10 to reduce the lag for motioncontrollers.

Until we have a way of accessing the HMD's final pose on the render thread we are reduced to hacking the engine.
Unless there a mechanism allready in place that im not aware of.
But like I said Epic seem to have a branch named orion!?!

I'll put the experiment up on github if you want to have a look?
The reduced latency definately helps with hand prescence.

That's an interesting approach, I'll have to look into some of the bind points you suggested. I know Motion Controllers get a late update function call, but I don't know if regular InputModules do too. There could be a way for the leap motion to pretend it's a motion controller module to get this late update without engine fix. That could be a fairly sizable refactor.

Regarding the branch, how do you find these branches? I haven't found a good way to browser UE branches on github. Sounds fascinating either way... not that I would know anything about that

[RalfRab]

Hello,

somehow I do not manage to see the leap motion image in my scene.
I have added a leap controller in my level BP and did enableImageSupport.

I have a cube in my scene with a billboard compontent. The sprite is scaled to screen size with factor 0.25.

I have added the key 1 event BP in order to see the leap motion image, but I actually is not showing up. Do you have any hints, what I might do wrong?

Thanks

RalfR

[Ders92]

Not sure about this one, first time I've seen this issue. You can try to correct world to meters using ipd instead (stereo e=<ipd value here>)

Again thank you for your answer it partly solved my issue as I wrote earlier. However the leap hands seems to get updated weirdly and not adapting to the change in "world to meters". So I looked through the source code for your plugin and found a function you use called "convertAndScaleLeapToUE" in LeapInterfaceUtility.cpp that is used when translating the hands. So my thought is that perhaps the worldToMeters variable should be used in this function so the hands can adapt to the change in scale.

I tried to change the leap hands translation like this:
FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * worldToMeters/100.0,
leapVector.x * LEAP_TO_UE_SCALE *worldToMeters/100.0 ,
leapVector.y * LEAP_TO_UE_SCALE * worldToMeters/100.0);

But this has no effect (tried with extreme values) and im suspecting its because I use blueprints and that the c++ files aren't connected to the blueprints? So my question is if there is a way to achieve this in blueprints or do I have to use your plugin with c++, or is there another way to fix my issue?

Thanks in advance!

[vimaxus]

I'll put the experiment up on github if you want to have a look?

Pretty pretty please? I for one would really need that and I can't find the time to try it (I wouldn't get it to work as well either).

[Opamp77]

That's very nice! How have you approached it? got any pull requests?

It involves engine mod's unfortunately.
Im using the same render thread delegate that I mentioned in the portal thread.
The delegate is an addition the the Headmounted display module.
A modifcation to the oculus plugin broadcast's to the delegate the final eye pose for each camera when it is usually calculated.

the bone's in this instance are staticmeshcomponents attached to hand actors.
At beginplay the LeapManager actor registers a callback function UpdateHandsOn_RT() with the delegate.

on GameThread Tick() the hand bones and update normally using the latest leap::frame at that point in time along with the results of GetHMDOrientationandPosition().

when the oculus plugin has calculated the final eye pose on the render thread it broadcast's location,rotation and Eye.
This fire's off the previously registed UpdateHandsOn_RT() in leaphandmanager.

This in turn calls ApplyLateUpdateTransform() on the sceneproxy's of each of the hands staticmeshcomponents using the latest availible leap frame along with the left eye pose.

I believe ApplyLateUpdateTransform() was implimented in 4.10 to reduce the lag for motioncontrollers.

Until we have a way of accessing the HMD's final pose on the render thread we are reduced to hacking the engine.
Unless there a mechanism allready in place that im not aware of.
But like I said Epic seem to have a branch named orion!?!

I'll put the experiment up on github if you want to have a look?
The reduced latency definately helps with hand prescence.

[getnamo]

Ive actually been playing with updating the hands on the render thread myself to get rid of some of the latency,

On an related note i've noticed that Epic seem to have an internal branch called Orion.
I wouldnt be surpised if there already working on it.

That's very nice! How have you approached it? got any pull requests?

[Opamp77]

From your video it seems you're using the official plugin, if you're using the unofficial plugin you can use the palm orientation value to drive the desires behavior.



Yep leap uses timewarp for their unity assets, its on the to do list but may take some time before I get to it. What they do is take a time sample along with the position and then grab the hmd orientation at the time. Then during late update they check the hmd orientation change between now and the timesample that was obtained earlier. The rotation difference is what is applied to make the timewarp.

If you manage to get it added make a pull request

Ive actually been playing with updating the hands on the render thread myself to get rid of some of the latency,

On an related note i've noticed that Epic seem to have an internal branch called Orion.
I wouldnt be surpised if there already working on it.

[Alllesss]

Hi, Thanks getnamo for ur awesome Plugin.
I'm trying to LineTrace at where Index Finger points at . just got confused on how to set it's Start And End Vector . sorry if it's not directly related ...