Announcement

Collapse
No announcement yet.

[Plugin] Leap Motion - Event Driven

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • replied
    Originally posted by getnamo View Post
    That's an interesting approach, I'll have to look into some of the bind points you suggested. I know Motion Controllers get a late update function call, but I don't know if regular InputModules do too. There could be a way for the leap motion to pretend it's a motion controller module to get this late update without engine fix. That could be a fairly sizable refactor.

    Regarding the branch, how do you find these branches? I haven't found a good way to browser UE branches on github. Sounds fascinating either way... not that I would know anything about that
    I just put my leap experiment up on github if you want to have a look. https://github.com/opamp77/UnrealEng....10Experiments

    Unfortunalty after thurther testing this does'nt appear to totally remove the rotational latency only reduce it but maybe im doing something wrong.

    As far as the Orion branch is concerned I noticed a few merges into the Master/4.11 branch after looking at a few more of them It appears that Orion is Epics internal name for Paragon.

    Looking ahead it looks like an applylateupdate() function is now being called within the hmd plugins.

    https://github.com/EpicGames/UnrealE...pplyLateUpdate

    It looks like in 4.11 you can inform the HeadmountedDisplay module of components that you would like updating on the render thread.

    https://github.com/EpicGames/UnrealE...etuplateupdate

    And it look like the camera gets a late update by default.

    If im right this should make things a bit easier.

    Leave a comment:


  • replied
    Originally posted by RalfRab View Post
    Hello,

    somehow I do not manage to see the leap motion image in my scene.
    I have added a leap controller in my level BP and did enableImageSupport.
    ...
    I have a cube in my scene with a billboard compontent. The sprite is scaled to screen size with factor 0.25.
    ...
    I have added the key 1 event BP in order to see the leap motion image, but I actually is not showing up. Do you have any hints, what I might do wrong?

    Thanks

    RalfR
    Make sure your sprite isn't set to hidden in game. While your bps look correct you can also try the EventRawImageReceived to get images.

    Originally posted by Ders92 View Post
    Again thank you for your answer it partly solved my issue as I wrote earlier. However the leap hands seems to get updated weirdly and not adapting to the change in "world to meters". So I looked through the source code for your plugin and found a function you use called "convertAndScaleLeapToUE" in LeapInterfaceUtility.cpp that is used when translating the hands. So my thought is that perhaps the worldToMeters variable should be used in this function so the hands can adapt to the change in scale.

    I tried to change the leap hands translation like this:
    FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * worldToMeters/100.0,
    leapVector.x * LEAP_TO_UE_SCALE *worldToMeters/100.0 ,
    leapVector.y * LEAP_TO_UE_SCALE * worldToMeters/100.0);

    But this has no effect (tried with extreme values) and im suspecting its because I use blueprints and that the c++ files aren't connected to the blueprints? So my question is if there is a way to achieve this in blueprints or do I have to use your plugin with c++, or is there another way to fix my issue?

    Thanks in advance!
    Still no better on this issue, I'm hoping someone else has a better understanding of what happens when you change scale. The thread for this problem is here: https://forums.unrealengine.com/show...ameter-problem

    Originally posted by Opamp77 View Post
    It involves engine mod's unfortunately.
    Im using the same render thread delegate that I mentioned in the portal thread.
    The delegate is an addition the the Headmounted display module.
    A modifcation to the oculus plugin broadcast's to the delegate the final eye pose for each camera when it is usually calculated.

    the bone's in this instance are staticmeshcomponents attached to hand actors.
    At beginplay the LeapManager actor registers a callback function UpdateHandsOn_RT() with the delegate.

    on GameThread Tick() the hand bones and update normally using the latest leap::frame at that point in time along with the results of GetHMDOrientationandPosition().

    when the oculus plugin has calculated the final eye pose on the render thread it broadcast's location,rotation and Eye.
    This fire's off the previously registed UpdateHandsOn_RT() in leaphandmanager.

    This in turn calls ApplyLateUpdateTransform() on the sceneproxy's of each of the hands staticmeshcomponents using the latest availible leap frame along with the left eye pose.

    I believe ApplyLateUpdateTransform() was implimented in 4.10 to reduce the lag for motioncontrollers.

    Until we have a way of accessing the HMD's final pose on the render thread we are reduced to hacking the engine.
    Unless there a mechanism allready in place that im not aware of.
    But like I said Epic seem to have a branch named orion!?!

    I'll put the experiment up on github if you want to have a look?
    The reduced latency definately helps with hand prescence.
    That's an interesting approach, I'll have to look into some of the bind points you suggested. I know Motion Controllers get a late update function call, but I don't know if regular InputModules do too. There could be a way for the leap motion to pretend it's a motion controller module to get this late update without engine fix. That could be a fairly sizable refactor.

    Regarding the branch, how do you find these branches? I haven't found a good way to browser UE branches on github. Sounds fascinating either way... not that I would know anything about that

    Leave a comment:


  • replied
    Hello,

    somehow I do not manage to see the leap motion image in my scene.
    I have added a leap controller in my level BP and did enableImageSupport.
    Click image for larger version

Name:	begin_play.png
Views:	1
Size:	73.3 KB
ID:	1101652
    I have a cube in my scene with a billboard compontent. The sprite is scaled to screen size with factor 0.25.
    Click image for larger version

Name:	event.png
Views:	1
Size:	147.6 KB
ID:	1101653
    I have added the key 1 event BP in order to see the leap motion image, but I actually is not showing up. Do you have any hints, what I might do wrong?

    Thanks

    RalfR

    Leave a comment:


  • replied
    Originally posted by getnamo View Post
    Not sure about this one, first time I've seen this issue. You can try to correct world to meters using ipd instead (stereo e=<ipd value here>)
    Again thank you for your answer it partly solved my issue as I wrote earlier. However the leap hands seems to get updated weirdly and not adapting to the change in "world to meters". So I looked through the source code for your plugin and found a function you use called "convertAndScaleLeapToUE" in LeapInterfaceUtility.cpp that is used when translating the hands. So my thought is that perhaps the worldToMeters variable should be used in this function so the hands can adapt to the change in scale.

    I tried to change the leap hands translation like this:
    FVector vect = FVector(-leapVector.z * LEAP_TO_UE_SCALE * worldToMeters/100.0,
    leapVector.x * LEAP_TO_UE_SCALE *worldToMeters/100.0 ,
    leapVector.y * LEAP_TO_UE_SCALE * worldToMeters/100.0);

    But this has no effect (tried with extreme values) and im suspecting its because I use blueprints and that the c++ files aren't connected to the blueprints? So my question is if there is a way to achieve this in blueprints or do I have to use your plugin with c++, or is there another way to fix my issue?

    Thanks in advance!
    Last edited by Ders92; 03-02-2016, 11:03 AM.

    Leave a comment:


  • replied
    Originally posted by Opamp77 View Post
    I'll put the experiment up on github if you want to have a look?
    Pretty pretty please? I for one would really need that and I can't find the time to try it (I wouldn't get it to work as well either).

    Leave a comment:


  • replied
    Originally posted by getnamo View Post
    That's very nice! How have you approached it? got any pull requests?
    It involves engine mod's unfortunately.
    Im using the same render thread delegate that I mentioned in the portal thread.
    The delegate is an addition the the Headmounted display module.
    A modifcation to the oculus plugin broadcast's to the delegate the final eye pose for each camera when it is usually calculated.

    the bone's in this instance are staticmeshcomponents attached to hand actors.
    At beginplay the LeapManager actor registers a callback function UpdateHandsOn_RT() with the delegate.

    on GameThread Tick() the hand bones and update normally using the latest leap::frame at that point in time along with the results of GetHMDOrientationandPosition().

    when the oculus plugin has calculated the final eye pose on the render thread it broadcast's location,rotation and Eye.
    This fire's off the previously registed UpdateHandsOn_RT() in leaphandmanager.

    This in turn calls ApplyLateUpdateTransform() on the sceneproxy's of each of the hands staticmeshcomponents using the latest availible leap frame along with the left eye pose.

    I believe ApplyLateUpdateTransform() was implimented in 4.10 to reduce the lag for motioncontrollers.

    Until we have a way of accessing the HMD's final pose on the render thread we are reduced to hacking the engine.
    Unless there a mechanism allready in place that im not aware of.
    But like I said Epic seem to have a branch named orion!?!

    I'll put the experiment up on github if you want to have a look?
    The reduced latency definately helps with hand prescence.

    Leave a comment:


  • replied
    Originally posted by Opamp77 View Post
    Ive actually been playing with updating the hands on the render thread myself to get rid of some of the latency,


    On an related note i've noticed that Epic seem to have an internal branch called Orion.
    I wouldnt be surpised if there already working on it.
    That's very nice! How have you approached it? got any pull requests?

    Leave a comment:


  • replied
    Originally posted by getnamo View Post
    From your video it seems you're using the official plugin, if you're using the unofficial plugin you can use the palm orientation value to drive the desires behavior.



    Yep leap uses timewarp for their unity assets, its on the to do list but may take some time before I get to it. What they do is take a time sample along with the position and then grab the hmd orientation at the time. Then during late update they check the hmd orientation change between now and the timesample that was obtained earlier. The rotation difference is what is applied to make the timewarp.

    If you manage to get it added make a pull request
    Ive actually been playing with updating the hands on the render thread myself to get rid of some of the latency,


    On an related note i've noticed that Epic seem to have an internal branch called Orion.
    I wouldnt be surpised if there already working on it.

    Leave a comment:


  • replied
    Hi, Thanks getnamo for ur awesome Plugin.
    I'm trying to LineTrace at where Index Finger points at . just got confused on how to set it's Start And End Vector . sorry if it's not directly related ...

    Leave a comment:


  • replied
    Not sure about this one, first time I've seen this issue. You can try to correct world to meters using ipd instead (stereo e=<ipd value here>)
    Thank you so much! This solved it However I have to scale the world down a bit it seems, right now i need world to meters at around 20 000 so i changed the ipd to 0.00064, but then the leap hands get out of control and starts vobbling like crazy. It worked fine to set world to meters to 1000 and ipd to 0.0064 so I will just scale the world down or do you have any suggestion on how to solve the vobbling hand issue?. If anyone has the same problem and wonder how to set the stereo just right click in your level blueprint, write "execute console command" and then write "stereo e=<ipd value>" (as getnamo suggested) in the command field.
    Last edited by Ders92; 02-29-2016, 06:41 AM. Reason: added question

    Leave a comment:


  • replied
    Hey Guys I'm using this plugin in ue 4.10 and whenever I play my game my hands are swapped. My leap motion thinks i'm in desktop mode or something. I Have tried chaning this setting everywhere. Has anyone else experienced this or know a possible solution?

    Leave a comment:


  • replied
    Originally posted by Sycol View Post
    Hi, I'm the guy with the axis problem. I'm new with leap and with Ue4 so im very lost. I want to do something like this https://www.youtube.com/watch?v=am6Rt2toZZc
    but it looks like that: https://youtu.be/3apQNG_30qE [I have not much idea of what i'm doing] so i want to ask you if you can please, please show me how?
    Thanks for all.
    From your video it seems you're using the official plugin, if you're using the unofficial plugin you can use the palm orientation value to drive the desires behavior.

    Originally posted by vimaxus View Post
    I think the low latency feeling only comes from the "temporal warping bias" that is presented as an option. If I put it to 0 I see the same "lag" as in UE4. I thought about doing something like this for some time and now I have the motivation as it makes quite a difference.
    I plan to get the relative position from hands to headset and move them to where they were relative to the HMD x ms before.
    How can I get to do that at each update hands event so that it doesn't add-up.

    I know it's not very inteligible, I'll try to add a blueprint or some scanned sketches.
    Yep leap uses timewarp for their unity assets, its on the to do list but may take some time before I get to it. What they do is take a time sample along with the position and then grab the hmd orientation at the time. Then during late update they check the hmd orientation change between now and the timesample that was obtained earlier. The rotation difference is what is applied to make the timewarp.

    If you manage to get it added make a pull request

    Originally posted by Ders92 View Post
    Hello!

    I'm currently using the getnamo plugin for leap motion in unreal engine 4 in combination with occulus rift dk2 to make a game. The game will be played on a virtual table so the players should be able to reach everywhere on the map (it should feel like you are sitting around a real table). In order to achieve this I want to change the "world to meters" parameter in world settings under VR, but when i increase it to for example 200 (default is 100) the leap motion hands start behaving weird. A cross eye effect occurs and if I increase it even more the leap hands eventually disappears completely. One way to solve it is to keep the world to meters parameter as it is and just scale the game world down, but this resulted in issues with the navigation mesh as I had to scale everything down a lot.

    So my question is if this problem when increasing the "world to meters" parameter is at all connected to the leap motion plugin or is it some occulus/ue4 specific problem? Or am I missing something obvious here? Haven't found anyone with the same problem so im starting to suspect that it might be something fundamental im missing. It's my first time working with ue4, occulus and leap motion so everything is kind of new.
    Not sure about this one, first time I've seen this issue. You can try to correct world to meters using ipd instead (stereo e=<ipd value here>)

    Originally posted by dmorgan View Post
    what the easiest way to handle picking up an object with the leap controller? How to get collision events per hand? Thanks
    Made a video series on this topic, follow along to learn how to make e.g. a jenga game:

    Part 1


    Part 2


    Part 3
    Last edited by getnamo; 02-29-2016, 10:51 AM.

    Leave a comment:


  • replied
    what the easiest way to handle picking up an object with the leap controller? How to get collision events per hand? Thanks

    Leave a comment:


  • replied
    Thanks! I don't know I'm doing so I really appreciate the response.

    Leave a comment:


  • replied
    Hello!

    I'm currently using the getnamo plugin for leap motion in unreal engine 4 in combination with occulus rift dk2 to make a game. The game will be played on a virtual table so the players should be able to reach everywhere on the map (it should feel like you are sitting around a real table). In order to achieve this I want to change the "world to meters" parameter in world settings under VR, but when i increase it to for example 200 (default is 100) the leap motion hands start behaving weird. A cross eye effect occurs and if I increase it even more the leap hands eventually disappears completely. One way to solve it is to keep the world to meters parameter as it is and just scale the game world down, but this resulted in issues with the navigation mesh as I had to scale everything down a lot.

    So my question is if this problem when increasing the "world to meters" parameter is at all connected to the leap motion plugin or is it some occulus/ue4 specific problem? Or am I missing something obvious here? Haven't found anyone with the same problem so im starting to suspect that it might be something fundamental im missing. It's my first time working with ue4, occulus and leap motion so everything is kind of new.

    Leave a comment:

Working...
X