Gesture Tracker VR Help and Feature Requests

[a_prototype]

Hey Hunter,

thanks for the reply. Is there a way to try this out before committing to a purchase? Or does Epic offer Marketplace refunding if it will not work properly for me?

[hdelattre]

I’ve had one refund request, which was from someone in China who did not properly understand the purpose of the plugin when they purchased it. In this case Epic contacted me and gave me the choice to approve or deny it (I approved it). If you find the plugin isn’t what you expected and doesn’t work for your needs and you ask Epic for a refund I assume it will go through the same process, in which case I will grant the refund. I’m not certain about the details of when a refund is allowed though so I don’t want to guarantee anything (in case they flat out say no and I’m not involved in the decision).

[anonymous_user_b3548bea]

hi. trying to install the unreal game engine with hi. using a screen reader jaws for windows, and cannot install the launcher. i am logged in via the site and then asks me to login in the launcher installer, and cannot install. any help.

[a_prototype]

Sounds great! I’ll give it a go then

[a_prototype]

Hey Hunter,

I just purchased it but it seems that the only way to recieve the plugin is through the epic games launcher using the “install into engine” procedure. Clicking the button won’t do anything though, i guess that’s because i’m on MacOS. Is there another way to download and install the plugin manually?

[hdelattre]

Hey @a_prototype,

I’m sorry about that! I just updated the settings on my product page for the 4.18 version to include Mac in the supported versions so give the install another shot and see if it works this time. If you’re using an earlier version than 4.18 let me know which one.

For now you’ll also need to recompile the plugin since it’s compiled for Windows. Move the GestureTrackerVR folder from Engine/Plugins/Marketplace to [YourProject]/Plugins to do this. Also you’ll need to add “Mac” to the list of Whitelisted Platforms in GestureTrackerVR.uplugin to make sure it packages with your game when you build (I’ll add this myself in the next update so you won’t have to deal with it in future versions). If for whatever reason you still can’t install it shoot me an email and I’ll send you a download link until I work out with with Epic how to fix installing through the launcher.

I’ve submitted an update with Mac added to the whitelisted platforms to Epic and indicated that they should add Mac binaries so you should be able to download everything through the launcher without any hassle within a day or two.

[a_prototype]

It works through the Marketplace now. Thanks a lot for your support!

[mnrmja]

@hdelattre

I figured I’d make a Gesture Tracker VR support thread so I can have a place for feedback, public support and feature requests.

https://youtube.com/watch?v=0D1Iy3Nprik

FAQ:

Does this work with Oculus Touch, Leap Motion, Perception Neuron, etc.?

Gesture recognition depends only on the transform of the GestureTracker component itself and does not depend on any device specific functionality or API so it will work with any input device. As long as you attach it to some moving component that you want to track it will work.

Is C++ supported?

C++ is supported but discouraged. The component was designed to be used in blueprints and is not as easily setup in C++. To use the plugin in code add “GestureTrackerVR” to your project’s public/private module names and then include “GestureTracker.h”. I’m willing to provide help to anyone using C++ or BPs but I highly encourage you to use BPs if possible. Make sure to keep your GestureTracker’s parameters within the range of values they are clamped to in blueprints. Values outside of the clamp range are not guaranteed to work and may cause a crash (e.g. Gesture Resolution <= 0).

Are Mac, Android, iOS, etc. supported?

These platforms are not officially supported since I have not tested for them but the component should work fine and users have successfully used the plugin on all of them. If you’re building for one of these platforms make sure to check GestureTrackerVR.uplugin in the plugin’s folder for the WhitelistPlatforms list at the bottom. Add your platform string to this list (e.g. add “Mac” or “IOS”) if it is not already present.

How do I save/load gestures? Can I package default gestures with my game?

Gestures can be saved and loaded using the GestureTracker component’s Save and Load functions. Gestures are stored in the Content folder so the arguments to these functions should be paths relative to the content folder. Gesture save files are just binary data files without type, but you can give them some standin extension if you wish. To provide a set of default Gestures with your game you can specify to package the gesture save with your game and load it when the game starts. To package the save file you’ll need to add the directory containing them to ‘Project Settings > Packaging > Additional Non-Asset Directories to Package’. So you would, for example, save all your gestures to a “Gestures” folder in the root of your Content folder and then add the Gestures folder to the list of Additional Non-Asset Directories to Package.

Can I perform multiple gestures at the same time?

Yes. You can add as many GestureTracker components as you’d like and they will all perform tracking and recognition independently. If you would like to do two handed combined gesturing you can just check that the GestureTracker on each hand recognized the desired gesture within a certain time frame of each other.

Does this have to be used with VR?

The system is designed with VR in mind but users have had success using it to track mouse and finger gestures (on mobile). Feel free to ask me about your use case if you’re unsure if this is right for you.

Changelog:

1.1 - 11/3/16 (4.13)

• Added baseMaterial input to draw functions. This is an optional replacement the draw mesh’s default material that lets you easily apply dynamic materials to the gesture draw.

• Implemented the GestureTracker Update Track/Predicted Draw Materials functions, as well as the Update Materials function for GestureMesh objects.

• Fixed Draw by Id and Draw by Name functions.

• Other minor fixes and comment improvements.

1.2 - 11/18/16 (4.14)

• Made GestureTracker.h public for C++ access. I still recommend using blueprints.

• Added GetNumGestures and GetIds functions to get the number of recorded gestures and an array of their ids respectively.

• Added UpdateTrackIds and UpdateDrawIds functions to let you change the blacklist/whitelist of drawing/tracking in the middle of recognition.

• Added RecognitionDuration float output to recognition functions and events which gives the time it took the user to perform the gesture in seconds.

1.3 - 2/22/2017 (4.14/4.15)

• Fixed an issue with tracked draw lines drawing incorrectly while moving and tracking by the world location.
• Fixed gesture disqualification tolerance so its the same for all resolutions (used to be so that a smaller resolution permitted a smaller off-gesture distance before disqualification).
• Added resetOnRecognized argument to continuous recognition. This lets you determine whether recognition for a gesture should restart as soon as it has been completed, or if it will only restart once the gesture has stopped being performed (e.g. If true, a single downward swipe gesture could be recognized multiple times by doing one long downward swipe. If false, you must stop performing the downward swipe and then start it again for it to be recognized a second time).
• Added GetGesturePath(int id) and GesturePredictedGesturePath() functions to get a TArray<FVector> describing the path of the gesture with the given id and the currently predicted gesture respectively.
• Removed unused EGestureTrackerMode enum.
• Minor naming and organization improvements.

1.4 - 6/2/2016 (4.16)

• Added BodyLocationComponent, a reference which is null by default. If set to point at your VR “body” (typically the camera object), the vector between this component and the
  Tracker will be used to determine that casting rotation. This resolves issues where the gesture rotation is incorrectly interpreted (e.g. if the player is pointing the
  motion controller forward but with a 90 degree twist in the wrist) but may also be slightly less precise about determining the gesture rotation. Leaving this null will
  leave rotation tracking working the same as it has in the past. This may or may not be better recognition depending on what kind of gestures/recognition are being used.

• Added trackFailDistance and continuousFailDistance parameters to determine how far off a given gesture’s path you must go to fail recognition of that gesture.

• Added predictionPercentage parameter that determines what percent a gesture must be completed before it can be displayed as the predicted gesture.

• Added rotationOffset parameter, which adds degrees to your gesturing rotation in case the GestureTracker is offset from the player’s actual aiming direction.

• Added GetPredictedGestureId() and GetPercentageComplete(int id) functions.

• Continuous recognition should now be more reliable. Expect more improvements in the future too.

• All parameters are now read/writeable in blueprints. NOTE: Please make sure you stick to parameter values inside the ranges of the sliders in the details panel. Values outside these ranges (e.g. gesture resolution <= 0) are unsupported and could cause crashes/other issues.

• Minor readability, naming, comments, and organizational improvements.

1.5 - 7/6/2017 (4.16)

• Added LengthRatio output to FinishRecognition and GestureRecognized events representing the ratio of the length of the tracked gesture to the length of the recognized gesture. So if the tracked gesture was twice as long as the recorded gesture it recognized then LengthRatio will be 2.

• Added NormalizePitchRotation to tracking params. Normally gestures are only normalized by yaw rotation. Enabling this will also normalize them by pitch so they rotate with you as you look up and down. This means if you define, for example, a forward punch gesture it will also be recognized if you punch up or down. Or if you draw a triangle in front of you it will also be recognized if you draw a downward facing triangle above you.

• Also added UseBodyLocationForPitch; if true we use the BodyLocationComponent for pitch normalization (if we’re normalizing pitch and the BodyLocationComponent has been set), otherwise we’ll use the gesture tracker rotation as normal. This should usually be false since, if you’re using your VR camera as the BodyLocationComponent, there will be a negative bias in the pitch (since your hands rest about 40 degrees below the camera).

• Moved GestureMeshComponent to the public folder to resolve an issue with blueprint nativization.

1.5.1 - 8/2/2017 (4.16)

• Added missing include whose absence could cause compile errors for shipping builds.

1.6 - 8/9/2017 (4.17)

• Updated to UE 4.17 but no meaningful changes.

Hii Hunter,

I am working on a VR Project which is entirely designed on Unreal Engine 4.12.5. I want to use your plugin in our project and implement it in the environment. So, I found out that Unreal Engine 4.12 doesn’t contain the VR Template. VR Template has been made available from Unreal engine 4.12. So, can you please help me with the further process to use the GestureTrackerVR plugin in our project.
Looking forward for a possitive response from your side.

Regards,
mnrmja

[hdelattre]

Hi @mnrmja

The VR template isn’t necessary to use the plugin, I was just demonstrating how it’s setup in the most basic environment. I assume your VR project has some hand actor (like BPMotionController in the VR template) that includes a MotionControllerComponent, just attach the GestureTracker component to it and do the same setup as I did in the video.

[jhonut]

@hdelattre

Hi@jhonut ,

I’m not certain that I understand your question but it sounds like you’re having a hard time getting your fireballs to fly upward instead of just horizontally when you do a gesture? You could just use the forward vector of your hand when the gesture finishes to define the movement direction of the fireball. More details or pics of relevant blueprints would be helpful!

Thank you very much it worked!!

[ForwardXPDev]

Hi, I’m having trouble finding where tolerances can be adjusted. Can anyone point me in the right direction?

Thanks!

[quickflint]

Hello Hunter,

I am interested in using your plugin. Is it possible to use two hands to create one gesture ID? Could I make a gesture only activate if both hands create circles? If that is possible, do they have to be done at the same time or could they be done a specific order? An example would be creating a circle with my left hand then pushing through it with my right.

[hdelattre]

@ForwardXPDev

Hi, I’m having trouble finding where tolerances can be adjusted. Can anyone point me in the right direction?

Thanks!

Hi @ForwardXPDev,

Check the details panel for the GestureTracker component and find Parameters > Recognition. You’ll want to tweak acceptable similarity (above which the gesture path progresses, this is the main one you’ll want to tweak) and unacceptable similarity (below which the gesture will be ruled out for recognition). You can also change the Completion Percentage if you want to allow recognition earlier on in the gestures (so you only have to do the first 90% of it for example. assuming the gestures aren’t distinguished by that last 10%).

[rharbin]

HAH! I would’ve sworn I checked there. Thanks for the help!

[hdelattre]

@quickflint

Hello Hunter,

I am interested in using your plugin. Is it possible to use two hands to create one gesture ID? Could I make a gesture only activate if both hands create circles? If that is possible, do they have to be done at the same time or could they be done a specific order? An example would be creating a circle with my left hand then pushing through it with my right.

Hi @quickflint,

I decided to leave that kind of functionality up to manual setup by users. You can just set up a gesture tracker on each hand, record your circle gesture with id 0 for each of them, and then check if they happen in the right order and time frame. So I when the left hand circle is recognized I would record a timestamp using GetWorldTimeSeconds, and whenever the right hand push is recognized I would check if (GetWorldTimeSeconds - CircleGestureTimestamp) < TimeWindow, where TimeWindow is something like 1 or 2 seconds that you have between drawing the cirlce and pushing through it.

[quickflint]

@hdelattre

Hi @quickflint,

I decided to leave that kind of functionality up to manual setup by users. You can just set up a gesture tracker on each hand, record your circle gesture with id 0 for each of them, and then check if they happen in the right order and time frame. So I when the left hand circle is recognized I would record a timestamp using GetWorldTimeSeconds, and whenever the right hand push is recognized I would check if (GetWorldTimeSeconds - CircleGestureTimestamp) < TimeWindow, where TimeWindow is something like 1 or 2 seconds that you have between drawing the cirlce and pushing through it.

Thank you! That is awesome!
I will be grabbing the plugin. Really happy to see you are still active in the support forum. Thank you very much for your response.

[ForwardXPDev]

Hi, I’m trying to save and use gestures for both controllers simultaneously (UE 4.19.2). I’m hitting a few different snags on this.

1. I can’t successfully save both sets. Some gestures will show up in one file but not the other. Some don’t save at all. Here’s my record/save/load set up:
   I’ve also tried using one each of the Start/Finish Recording nodes with both trackers plugged in as the target. I get the same result either way.

2. I don’t know how to get the below nodes created. I assume I need one to recognize the Left controller but I can’t find how to create these nodes. They do not show up in the right-click menu.
   

Thanks!

[hdelattre]

Hi @ForwardXPDev ,

1. Each tracker saves its own set of gestures so you shouldn’t expect any from one file to show up in the other. Based on your setup my best guess for why some gestures are disappearing is that maybe you’re recording to the same Id multiple times, which overwrites the gesture previously recorded to that Id. Also, gesture recording fails when the recorded gesture’s length is greater than the maximum allowed length (you can tweak this in the tracker’s details panel), so I suppose it’s possible you’re recording very long gestures sometimes and they’re failing. Your setup looks right (although the delay between saves is probably unnecessary) so I’m not sure what the issue would be otherwise.

2. If you click on one of the GestureTracker components and look at the bottom of the details panel you’ll see some events with ‘+’ symbols next to them. Click these to create the events shown above. Alternatively you can drag a pin off a tracker and call BindToGestureRecognized, then drag off the resulting bind node’s delegate pin and create a custom event (this will auto generate an event with the proper signature, or inputs).

[calvin019]

Can u get the speed of how fast the tracker completes the pattern? or if it only completes 50%?

[hdelattre]

Hi @calvin019,

The Recognition Duration output on the gesture recognized event tells you how long it took (see ForwardXPDev’s images above).

If you want to recognize gestures at 50% you can lower the Completion Percentage parameter on the GestureTracker to .5. I may add support for setting individualized completion updates in the future (e.g. have the tracker tell you when a specific gesture id has reached a specific percentage such as 50%). As an alternative for now you could record a gesture that’s the 50% version of the full gesture and then check for that from the GestureRecognized event, which will fire for both the 50% gesture and the 100% gesture if both are performed during recognition (instead of just using the result from FinishRecognition, which only cares about the final gesture you performed and doesn’t fire until user input tells it that the gesture is done being performed).