Download

Mobile Gesture Recognition Toolkit

As far as I can tell, there is no built in gesture recognition within UE4, meaning simple functions that return wether a single tap vs a double tap occurred and where, wether a ping to zoom or two finger scroll gesture was used and how much it moved. I would think it would make all mobile developer’s lives better if they didn’t have to each create their own versions of these functions, not to mention helping to standardize how people interact with mobile projects.

That functionality comes built in for many other, more mobile-centric engines, so I am surprised it doesn’t exist at all what is considered the premiere game engine out there. I’d like to request that this gets investigated and a response given as to wether Epic will be adding this functionality in the future, and if not, why not.

Cheers,

J^2

You know, I have to throw a +1 on this idea. I’ve been toying with the idea of writing one myself for Republic Sniper. We’re currently not using gestures, but I’d like to add some gesture support. I’ve got a lot of old pre-Gesture Recognizer code from iOS that probably wouldn’t take too long to port over.

Won’t get a chance to look at it for a few weeks. If I go ahead, I’ll make it a public project others can use or contribute to. If someone else beats me to the punch, I’ll contribute to theirs when I get to it.

There is FGestureRecognizer which detects gestures as part of the PlayerInput Tick and it (in theory) will identify 6 gestures and submit them as an FKey that you can bind anywhere you can normally bind a key (Action Mapping, On Key). However, it is admittedly not in great shape at the moment and I believe that several of the gestures are not even detecting properly :(. It is also not extensible which I think is probably necessary. It is in the backlog to clean up and get working better, but if you were looking at investing some time on gestures in the near term and wanted to use it as a starting point it might be something that would be able to be integrated back in to the engine via pull request.

Marc:

Thank you for that information - I was not aware of that class.

And, I happen to be stuck on a flight with plenty of time to kill and a copy of the engine code on my machine (though, not sure how up-to-date it is)

Had some down-time at GDC, started expanding the gesture support in the engine. Are there additional gestures besides these that you think the engine needs?



    // Gestures - Swipes
    static const FKey Gesture_SwipeLeftRight;
    static const FKey Gesture_SwipeRightLeft;
    static const FKey Gesture_SwipeUpDown;
    static const FKey Gesture_SwipeDownUp
    static const FKey Gesture_TwoFingerSwipeRightLeft;
    static const FKey Gesture_TwoFingerSwipeLeftRight;
    static const FKey Gesture_TwoFingerSwipeUpDown;
    static const FKey Gesture_TwoFingerSwipeDownUp;
    static const FKey Gesture_ThreeFingerSwipeLeftRight;
    static const FKey Gesture_ThreeFingerSwipeRightLeft;
    static const FKey Gesture_ThreeFingerSwipeUpDown;
    static const FKey Gesture_ThreeFingersSwipeDownUp;
    static const FKey Gesture_FourFingerSwipeLeftRight;
    static const FKey Gesture_FourFingerSwipeRightLeft;
    static const FKey Gesture_FourFingerSwipeUpDown;
    static const FKey Gesture_FourFingerSwipeDownUp;
    static const FKey Gesture_SwipeUpAndRight;
    static const FKey Gesture_SwipeUpAndLeft;
    static const FKey Gesture_SwipeDownAndRight;
    static const FKey Gesture_SwipeDownAndLeft;
    static const FKey Gesture_SwipeFromLeftEdge;
    static const FKey Gesture_SwipeFromTopEdge;
    static const FKey Gesture_SwipeFromRightEdge;
    static const FKey Gesture_SwipeFromBottomEdge;
    
    // Gestures - Taps
    static const FKey Gesture_Tap;
    static const FKey Gesture_DoubleTap;
    static const FKey Gesture_TripleTap;
    static const FKey Gesture_Long_Tap;
    static const FKey Gesture_TwoFingerTap;
    static const FKey Gesture_TwoFingerDoubleTap;
    static const FKey Gesture_ThreeFingerTap;
    static const FKey Gesture_ThreeFingerDoubleTap;
    static const FKey Gesture_FourFingerTap;
    static const FKey Gesture_FourFingerDoubleTap;
    static const FKey Gesture_FiveFingerTap;
    
    // Gestures - Other
    static const FKey Gesture_Pinch;
    static const FKey Gesture_Flick;
    static const FKey Gesture_Rotate;
    static const FKey Gesture_Touch1Pan;
    static const FKey Gesture_Touch2Pan;
    static const FKey Gesture_Touch3Pan;
    static const FKey Gesture_Touch4Pan;
    static const FKey Gesture_Touch5Pan;


I know that touches are easy enough to get, so the [FONT=Courier New]Gesture_Tap may seem unnecessary, but I still think they should go through the gesture recognizer so, for example, double-taps don’t also send a single-tap message. There are times when you want to know every touch, but other times when you just want to know if there was a tap (that wasn’t a part of a double/triple tap).

If I might throw in a couple more gestures, perhaps expand rotate to both 2 and 3 finger rotation. Also perhaps expand pinch to pinch in and pinch out? Maybe even 2 finger pinch and 3 finger? The more options there are the more creative the interface can get. Thank you so much for taking the time to do this. Will you be by chance be adding a blueprint node for this as well for us codeophobes? Cheers,

J^2

I’ve never seen a three-finger rotate gesture used in an app, but that would be easy enough. Same with three-finger pinch.

My intention is to do this as an Engine patch, so they should all be bindable to events in Blueprint. I’m still studying the way gestures are implemented, though, so I haven’t come to a final determination about how I should do it. There are some limitations to the way it’s currently implemented, and I might make the gestures their own custom events outside of the ordinary input system. I don’t like, for example, that the current pinch gesture gets sent every tick, even after you’ve lifted both fingers of the screen, for example. Changing this to the way I think it should be implemented based on my experience in mobile would totally break backwards compatibility, so it’s going to take a little though to do this right.

I think making a system that is outside the normal input events is probably the way to go.

For the really basic ones keeping both a EKey and EGesture (or whatever) version might make sense so you can bind just a 1 finger swipe to normal input events but get no useful information or bind it as a gesture and get whatever information is appropriate for that action.

I was pondering this the other day and I was thinking a slick way to do this would be to have a Gesture system that allows new gestures to be registered (similar to EKeys) along with the delegate type it will implement. InputGesture could potentially work similarly to InputKey where you can pick between all the defined gestures and when the Gesture is selected the node would automatically change the exposed pins for that particular gesture, but it may just be simpler to make it static based on the gesture it was placed with. Then the gesture recognizer, when it dispatches the event, can supply the custom information appropriate for that event.

Thanks for the input. Curiously enough, I came to the same conclusion a few minutes before I got notice of your reply. I’m creating an [FONT=Courier New]FAdvancedGestureRecognizer class that will get ticked from a [FONT=Courier New]UPlayerInput subclass used by our player controller. I’m currently working on defining an interface for gesture delegates now. Unfortunately, Unreal doesn’t make it easy to create a C++ interface, so I’m fighting the build system a bit. “Add Code to Project” doesn’t have an option for subclassing [FONT=Courier New]UInterfaces. I created one manually based on Rama’s Wiki page entry, but the Create Xcode Project service isn’t creating a [FONT=Courier New].generated.h file for it, which the build system is saying it has to have. :frowning:

To some extent, I want to model this after the iOS [FONT=Courier New]UIGestureRecognizer, but there are some challenges involved in doing that. In a regular non-game app, it’s okay to wait to see if you’ve gotten a double-tap before firing off the single-tap delegate call. In a game, that half-second or second delay will generally not be acceptable. However, if you support double-taps, you probably don’t want a double-tap to fire both a single- and double-tap delegate call. We generally use “touch down” for most of our stuff for responsiveness, but we can’t exactly call back a bullet after it’s been fired. That means gestures are going to be a challenge.

Yeah, that would offer some nice flexibility to let people define their own delegates. I’d like to structure it that way, but not sure I can justify the time, since our game doesn’t need that level of flexibility.

After a bunch more experimentation, I’ve got a proof-of-concept implemented as actor components using delegate calls. There’s a virtual base class, [FONT=Courier New]UGestureRecognizerComponent, that keeps track of information about each touch in the gesture. Concrete subclasses use this data to detect if a gesture has happened or is happening.

The delegate calls return a reference to the gesture recognizer, which can be queried for whatever information is needed. A rotation gesture recognizer can be queried for delta angle, for example, or a tap gesture can be queried for how many fingers were involved in the tap, or how many taps there were.

I’m feeling pretty good about this approach, so far. It’s fairly flexible and modular and only requires processing to look for gestures you’re actually interested in. Any gesture you don’t add a recognizer for doesn’t get ticked because it doesn’t exist. Creating new concrete subclasses is relatively easy.

The only thing I don’t like right now is that the gesture recognizer is assuming that it’s been added to an [FONT=Courier New]ACharacter subclass whose controller is an instance of [FONT=Courier New]APlayerController. The reason for that is I wanted to leverage the touch processing that’s already happening in [FONT=Courier New]UPlayerInput. It doesn’t break if it’s not, but it fails to detect any gestures.

Is there a better way to handle this? I looked at creating a subclass of [FONT=Courier New]UPlayerInput, but [FONT=Courier New]Tick is not virtual in that class. Alternatively, is there a way to specify that the component should only be added to [FONT=Courier New]ACharacter instances?

Heh. There’s nothing like asking the world about something to make it obvious. I can get the [FONT=Courier New]UPlayerInput by getting the local player controller.

Okay, made some progress. For anyone interested in seeing how this is coming together, I’ve pushed my project up to github:

So far, I’ve only got the tap and swipe gesture recognizers done, but I’ve also got a lot of infrastructure in place for doing the rest of them fairly easily. The Gesture Recognizers themselves are in C++, but the sample project uses them from Blueprint.

The basic process is:

  1. Add a gesture recognizer component to an actor. In the sample project, it’s the player’s Character blueprint
  2. Configure the gesture recognizer to identify the types of gestures you want detected
  3. Add the GestureEnded delegate event to your blueprint (Green button)
  4. Do whatever you need to do in your blueprint. If you need more details, you can query the gesture recognizer. In the sample project, the blueprint queries for the number of taps and the type of swipe.

It’s still early and rough, but I think it’s a good approach. Feel free to check it out and provide feedback.

Outstanding! I’m assuming I would have to be a bit more code saavy to grab and build your example so I will have to wait, but it sounds fantastic. I can’t wait to try it out. Thanks again for doing all this. Cheers,

You do need to have Visual Studio or XCode installed for it to compile, but otherwise, there’s actually no need to do anything in C++. The project will compile the code automatically on launch, and these recognizers can be used (and, really, are designed to be used) 100% from Blueprint.

Since my last posting, I’ve added a pan gesture recognizer and started a pinch/rotate recognizer. Unfortunately, I’m having some trouble with the rotate algorithm. I’ve used a couple of different algorithms for determining the change in angle between ticks (one a direct port of code that’s in many shipping iOS apps), but I keep getting weird results. Hoping a weekend away and approaching it with a clear head on Monday will help, as I wasted a lot of time on Friday with it.

Is there a way to distirbute a project with C++ without requiring people to install VS or Xcode to use it?

Hey, how can i recompile this for use in 4.9?

Well, it should work in 4.9 without recompiling unless they removed or changed the signature of an API call I use.

If they have, then you’ll have to set up a code project with Visual Studio or Xcode. When you launch a code project, it will automatically find the plugin code and recompile.