Speech Recognition Plugin - Sphinx-UE4

Looks a little choppy because 1) mouth shapes are just my interpretations of the visemes into Fuse’s morphs, so they’re not that accurate and 2) blending between morphs in my project is currently a bit hacky and 3) it’s not taking into account how long the phonemes are, it’s just doing each the same amount of time.

I tried your plugin and it works) but strange, i tryed 3 words: “spawn, one, two” in dictionary mode. All the time he recognizes “two”, what’s wrong? And phoneme mode doesn’t work at all.

You were able to make it work? I kept getting “speech recognition not compatible”. I used the 4.11 dll…

Pr0t0Ss12 , from the wiki:
"At the moment, this plugin should be used to detect phrases. (eg. “open browser”). Singular words recognition is poor. I am looking at ways to improve this to a passable level."
Ignore the Phoneme mode for now.

I have noticed single numbers are especially poorly recognized.
I suspect much of this is relates to issues with Pocketsphinx.
For example, the word “two” in the dictionary is made of two phoneme’s.
From =short&s]=words"]CMU Sphinx Tutorial](Building a language model – CMUSphinx Open Source Speech Recognition)

“For the best accuracy it is better to have keyphrase with 3-4 syllables. Too short phrases are easily confused.”

If you feel you can make improvements to the accuracy, by all means, download the source and get working on it. :slight_smile:
It’s available on GitHub. GitHub - shanecolb/sphinx-ue4: A speech recognition plugin for Unreal Engine 4. This is essentially a
Please let me know if you make some significant progress.

thewolfgoddess, the 4.11 binaries were built against the preview releases of 4.11.
I see now that the main release build is available, so it’s possible the previous binaries are not compatible.
Later tonight, I`ll download the main release build of 4.11, and investigate.

Ah, alright that makes sense. I rolled it back to 4.10 and got it working. I can try to work on the accuracy if I am able, but at the moment I am trying to finish a project before the deadline XD I have noticed that it works better with phrases, or it goes a little wonky, but I am now having a different issue… Is this plugin only compatible with Speech to text? I have been trying to get it to trigger actions and events but it only seems to work when using it to output strings. Otherwise it just does… nothing. I am probably doing something wrong lol

I have added a section to the wiki:

Please take a look. :slight_smile: and let me know of any problems.
I have created an experimental branch in Github. (this is what is used by the above demo)
For now, the major changes are correcting the enum spelling + more granular tolerance settings.
However, in the future, I plan to change this up considerably, and merge back when it is significantly improved in all respects.

Tomorrow, I plan to create some C++ examples, that will show-case the plugin as well.

A number of people had asked how to use my plugin within C++, instead of just blueprints.
I have updated the code (experimental branch), as well as updated the example projects to include an example.
Please take a look, and let me know if there’s any serious issues

My HTC Vive arrived about a week ago, otherwise the Android port will probably have been done by now.
Hopefully I can focus on that this weekend.

Yay for Android port! :slight_smile:

So I’m not sure why I had some good results with the phoneme recognition and then it later went to crud, but I’m going to start testing this again soon and see if I can get you a version that gives the results I saw before.

No problem. I think I recall reading that their phoneme recognition wasn’t great :confused: by their own admission.
Let me know if you make any headway.
I`m going to look at adding a Grammar support, and have it switching based on trigger-able contexts and such.

Would it be better (and hopefully easy) to utilize Android’s native speech recognition on Android ?

motorsep, yes, I am definitely looking into the possibility of using the Android native speech recognition.
I will keep everyone posted.
If anyone takes the existing plugin, and makes any noteworthy changes to the plugin, please don’t hesitate to contact me :slight_smile:

Just wondering how it’s going with Android version :o

Poorly, not because of having tried, and failed…simply because my Vive came, and all my spare time programming/UE work has been spent on that.
I`ll sit down and work on it solidly shortly :confused: Sorry.

Oh, ok. Understood and thanks.

Would it be possible to analyze sound files and generate phonemes with this plugin? I’d like to drive a lip-sync rig with those phonemes. If it is possible can you point me in the right direction to get started with solving this problem.

If you have a read earlier in this thread, both myself and n00854180t have experimented with phoneme detection.
However, the results have been wildly unreliable, even with their test application (outside of UE).
If you’d like, I can send you the sample code, to have you see for yourself/experiment?
Are you capable of recompiling the plugin? It’s pretty simple.
All you really need to do, is right click the .uproject file (of a project with the plugin), then regenerate project files.
Then open the project in visual studio.
The plugin source should now be visible, and you should just be able to build the project.

How do I get the defected phonemes form shinx?
A output like the following would be awesome:

G 0.3 0.5
OW 0.5 0.7
F 0.6 0.9

First of all, thank you so much for this Plugin it is great to have something like this available.

I have been able to get the plugin working in unreal 4.11.2 (Though I am having a lot of difficulty with accuracy of phrases)

I am however having a lot of issues when it comes to packaging the game. Here are the things I have found so far:

  1. Cannot get the plugin to work with a blueprint only project. This seems to be something that Epic is aware of, but haven’t done anything about. Only solution: Add a C++ class to the project so it builds it as such and includes all plugins.
  2. Must package for Windows 64 bit, trying 32 bit will fail the build process mentioning that it cannot access SphinxBasex86.lib (it doesn’t exist)
  3. Once the game has been packaged successfully it still will not launch from the executable. In order to get it to launch I must go find two .dll files (pocketsphinx.dll & shpinxbase.dll) and put them into “WindowsNoEditor\PROJECTNAME\Binaries\Win64”

At this point the game will launch and is completely playable. However, Init is failing :frowning: So even after all that I still have no speech recognition in a packaged version of my game.

Any ideas?

Hello JTensai,

Sorry for the late response.
Regarding your issues with the accuracy, can you please send me a PM including the following

  • Listing of the phrases you are attempting to recognize, along with the tolerance settings for each.
  • Audio recording of the phrases being spoken.
    Then I can take a look.

Yes, unfortunately this is the case until there’s a change to UE4. I have read recent posts by others who have the same issue (Plugin not packaging in Blueprint only project).
I have added a note to the Wiki that a empty C++ class must be added to the project, in order for the plugin to build.

My mistake, I had only included the 64 bit lib’s and dll’s of the Sphinxbase and Pocketsphinx.
I have updated Github Code/Sample Projects to reflect the changes.

This should be fixed by my latest check-in to the Git repo.
The dll’s are now copied during packaging.
Note, you have to add “model” as a folder to copy during the packaging process, otherwise the language model is not found (probably what was happening in your case, after you copied across the dll’s).


Let me know if you still are encountering issues.