Hello all,
I have ported my speech recognition plugin to Android, and decided to make it open, rather than a paid plugin.
This is an early demo to test whether the functionality is working for others, and hopefully fix-up any instructions before I update the wiki etc.
Since this a WIP, please try the demo project, and ensure this works appropraitely, before trying to use it within your project.
Let me know if you encounter any issues in trying to get it to run.
Only use Keyword recognition mode, the grammar mode for mobile appears somewhat broken.
It appears phrases which violate the grammar rules, still get recognised if they partially match.
Eg. For the following grammar, saying just “two” or “nine”, will return as a match, despite it not being of the form <digit> <operation> <digit>
Currently, only packaging/installing a project allows speech detection to work.
Launching a project directly will cause the project to crash.
This is because the speech recognition models are copied and extracted from the .obb, which does not happen when the project is launched directly.
I do think it would be possible to manually copy the speech recognition models to an external location, and update the references to point to them each time, but I have yet to try this myself.
Changes:
OnWordsSpoken now returns a collection of FDetectedPhrase’s (in the order of which they are recognised).
This allows a single call to contain all phrases that were detected in an utterance.
This enables logic to implement multiple phrases/commands at once (previously OnWordsSpoken would have been called twice).
The FDetectedPhrase contains more than just the phrase. Start/End time of the detected phrase, so the length of a phrase can be calculated, and used.
There is also an avgVolume property, but for now, I am not setting it. Future version hopefully.
Tolerance is now an Integer (rather than a enum setting). This makes more sense, and allows more flexibility.
Start with 30, and work up/down in increments of 5 or 10, to find the best balance between no detection and misfires.
How to test the demo project:
Extract the archive project to unreal engine project path.
Open the project file, and hit File=>Package Project=>Android=>Android Multi (unless you know your intended architecture)
Deploy onto your device, and run.
The default map is a set of cubes. When the player colides with a cube, a test map is loaded.
I would suggest running the game map, since it’s an example how actions can be controlled by the recognised phrases.
The other maps are examples of how the plugin can be used by only C++, or as a test of alternative implementations (alternative languages, grammar/language modes).
How to implement within your project:
Ensure you can build/run your project successfully on your android device.
From the demo project, copy the following folders to your project
Anyone…anyone at all? It’s funny how quite a few people were asking for an android port, and now it’s done, no-one wants to test it haha shrug.
Maybe i`ll just put the info on the Wiki, and if someone has issues, they can e-mail me.
It just would just be nice for a few people to test, to ensure the instructions work correctly for other people as well.
Can you try opening the project with 4.18? any number of issues can occur when trying to upgrade the project to a later version.
When I get some time free, I`ll get a 4.19 compatible project tested, and uploaded.
Oh, I can open it with 4.18 no problem. It’s just my project (that compiles and runs on Gear VR) is for 4.19 and UE4 refuses to open 4.19 project with 4.18 Editor
I was able to build it in 4.18 and it launches, but for whatever reason it’s asking me to setup store key, even though I am making dev build (I never had to set up store key for dev builds, only for release builds)
EDIT: Oh, I see what’s going on - it can’t find OBB and trying to download it from the Play Store… And for whatever reason I can’t packaged it into single apk, which might be a problem for Gear VR / Oculus Go - I recall they can’t have OBB files.
EDIT2: I installed it using adb (bat file UE4 generated when building). It launched, but not sure what to do there
Basically when going into C++ example, nothing happens until I say “hello”. After that I see print out on the screen, something like “init successul” and “hello world” on the next line.
Game example says game over as soon as it’s loaded.
I wouldn’t test the grammar example, the grammar is broken as far as I can tell in the android port (as I mentioned in my first post).
That being said, it shouldn’t crash, if the other maps don’t crash, it should just not work correctly…
Are you able to clear your android log, run it until the crash, and then generate a log?
Run “adb logcat -c” to clear the log, and “adb logcat -d > log.txt” to log the debug log to a text file.
The language example is just a demo map, for testing different languages, ignore that.
You really should just look at the Game map, the others were just for examples of alternative implementations.
The C++ example only has “hello world” and “open door” as recognizable phrases.
The only thing this map does is print the text to the screen when the phrases are detected.
I find it odd that the C++ example prints “hello world”, but the game map doesn’t recognize speech.
I`d expect either both to work, or none at all. The map says game over at the top when it loads initially, since no game is active.
The recognised phrases should still trigger.
Here’s a video of it in action, as you can see, it starts with Game Over at the top.
Ah, well, I didn’t know the commands for the game mode Tested - works the same way as on your video (on my side it was slightly more laggy than on your video).
It appears the 4.19 release isn’t as straightforward as I had hoped, but i will put it up in the next few days, once i get some time to sort it out.
Adding multiple phrases, which contain part of another phrase, is not necessary, and can cause issues.
The keyword recognition mode will match “walk” every-time that “enable walk” “start walking” is spoken.
“walk” is also very phonetically simple, and would likely cause problems (eg. false positives may trigger).
From the pocketsphinx FAQ:
“For the best accuracy it is better to have a keyphrase with 3-4 syllables. Too short phrases are easily confused.”
Not really, haven’t had any time spare, sorry.
Between my main day to day job, and the contracting work i am doing on a game, I haven’t had any spare time to look at it.
I`ll try and find some time this upcoming week.
I had some issues, because I originally thought the crashes I was getting was related to my project.
After some testing, even basic projects would fail to run on my tablet (details here: https://forums.unrealengine.com/deve…en-is-crashing).
In the end, when I tried the project on my Samsung S7 phone, it worked as if it were a 4.18 release:)
Hi Shane, Thank you for all your work in supporting this plugin. Will the Android version work for UE4.19 Windows builds? If not is there a windows 4.19 version available? (The demo projects on the wiki only go up to 4.18)
Edit: Used the 4.18 plugins for my 4.19 build and they appear to work just fine. Problem resolved.
@ShaneC
Amazing efforts sir!
The project works perfectly on windows, however when built it up to my Galaxy S8 the demo runs it doesn’t listen to the voice commands, what could be the reason?
Besides, will your plugin support IOS in the future by any chance?
And thanks again for the huge contribution.
Tested - works the same way as on your video (on my side it was slightly more laggy than on your video).