Download

Sphinx-UE4: Speech Recognition Plugin [Release v1.0]

Hello everyone!
I have been working on a number of big changes and fixes for my Speech Recognition plugin.
I am looking for some feedback from the Community.
This will hopefully become the official 1.0 release.

Please have a read of the Wiki, for demo project/plugin link:

Let me know if there’s any issues.

Changes:
OnWordsSpoken returns an array of detected phrases:
Previously, a string would be returned that would represent all of the phrases that were detected.
This made it tricky to process, and distinguish phrases (especially if multiple were spoken).
BUG FIX: Ensure word recognition is ordered:
When multiple phrases were spoken, they would not always appear in the same order as they were detected.
The array of detected phrases is now always ordered.
Added Grammar support​:
This allows JSGF files to be used as an alternative to Keyword matching mode.
**NOTE: **This is experimental. Demonstration of the functionality can be seen in the example blueprint. (Key events
‘A’,‘N’). From my experimentation, this mode attempts to match very poor matches.
So, you could say blah, and it match to “five”, or something.
In future, i will look at ways to ignore poor matches.
Added Mac/OSX support:
My OSX skills are pretty basic, so let me know if you encounter any problems with Mac usage of the Plugin.
NOTE: ​I have made some code changes, and have yet to jump back on and re­test on the Mac.
Allow Sphinx parameters to be passed in dynamically:
Previously, the parameters that are used to initialise Pocketsphinx were hardcoded. If you wanted to change the
sphinx parameters, you had to modify the plugin source, and recompile.
Now, you can set them dynamically, on the fly. (example in the demo project)
Added support for words with Multiple phonetic listings (when working In Keyword mode):
For some words, the dictionary lists multiple phonetic dictations. Eg. (absolve AH B Z AA L V
absolve(2) AE B Z AA L V). Only one could previously be used. Now, you can use both, and have only 1 phrase
be reported. However, to do so, you must include both phonetic spelling of absolve (eg. absolve and
absolve(2)

w00t! Congrats on 1.0 release! :smiley:

Hopefully we’ll see Android support soon :o

What does this do exactly? I have a lot of dialog in my game.
is it a speech engine that processes text and plays it in the game so you don’t have to
write sound files?

That would be a text to speech engine, this plugin allows you to detect spoken phrases, to trigger game actions.
Here’s a video I made a month or so ago, showing it in action.

@ShaneC - the keyword recognition seems quite reliable now.

For the grammar modes, I tried to test them based on what the grammar files had, but was probably doing it wrong.

Hello n00854180t,
Thanks, I really want to get this working as reliably as I can.
There’s still work to do, i just hope I am going in the right direction. :slight_smile:

Ahh, I think I see the issue with the Grammar mode…it’s only logging to the output log, and not adding to the set of recognised phrases
Try the following at ~ line 488. I have also updated the demo project link, with the fix.

Try the following:

-Start the demo project, and open the Output Log.

  • Hit ‘I’, then ‘N’.
  • This activates the following Grammar file:

Essentially, it tries to find a match of the form <digit> <operation> <digit>

If you say something like “two add four” , then the Output Log would report:

Note: At the moment, it appears the grammar format does not enforce that all 3 must exist.
For example, saying “two”, will trigger as “two”, rather than finding closest match that fits the form <digit> <operation> <digit>
I`ll look into whether I can enforce this, there’s very likely a way.

I’ll give it another test tonight and report back.

Really awesome!

Do I have to do something other than just adding a word to the dictionary in order for it to be callable? Do I have to rebuild the plugin or should I add anything more. It’ appears as when I use the “Mak RecognitionPhrase” with my word and add it to an array with existing words, not even the existing words will work. But if I remove my word from the array, the other words work just fine.

Thank you!

//Joakim

It seems my code did not correctly set Sphinx params.
I have committed an update to GitHub.

I will update the sample project in the coming days.

For sphinx params, I have had pretty good results with the following:

**
-vad_postspeech 10
-vad_prespeech 10
-agc “max”
**

I`m sure I can improve my accuracy, with additional tweaking, it’s just 3AM atm :confused:

My game has got nearly 15,000 text dialog segments to it. so it would need to be read from a
database of text ID strings and flags and speaker name. it’s so easy to set it up in windows script language, but hard to do in Unreal. Blueprints are limited. You need a text string database.

Without some context, that made little sense to me.
The methods and classes can easily be called through c++, if you wished to do that instead.

Sort of like this

Speaker ID: Storm0391…“String Text Dialog”…%flag name%…%Flag Counter%…%Audio file pathname%

The Problem is, I don’t understand C++
so i have a huge disconnect between the language that Unreal Uses and the language that I know which is simple script.

In other words you need a unreal coder to decipher my script language into unreal code.

I still don’t understand what you are trying to achieve.
From the fact you mentioned 15,000 text dialog in an earlier comment, it sounds as if you are looking to convert text to audio.
That is not the purpose of this plugin.

I have made a few small improvements/fixes to the plugin, and have created a tag for Version 1.0

https://github.com/shanecolb/sphinx-ue4/releases/tag/1.0

The Wiki has been updated, and includes a demo project (as seen in the video below)

https://wiki.unrealengine.com/Speech_Recognition_Plugin

https://youtube.com/watch?v=b6vd0pWSGKU

Just wondering if there are any news about Android support.

Unfortunately, my spare time has been spent on developing a HTC Vive input mapping application, controlled by javascript scripts (eg. Vive input controlling mouse, keyboard, game-pad).
The port is still on my to do list, but I first want to release a super early alpha of the Vive mapper, for people to try out, before I switch back to working on the Android port of the plugin.

hello ShaneC.
I’ve got some problem with it.
I want to get the current volume value,but it always prints 0.
So confused,Idon’t know where is wrong in my blueprint.

hello ShaneC.
I’ve got some problem with it.
I want to get the current volume value,but it always prints 0.
So confused,Idon’t know where is wrong in my blueprint.

First of all your plugin looks awesome and I would love to see how much it will grow.
I opened the project, then the game map, run the game and then when I tried to give commands from my mic, nothing happened.
I’ve been trying to find solutions all day.
Do I have to compile anything or do anything else before trying to run the project in UE4?
I need any help I can get, because I am really interested in seeing how this plugin works.

Thank you very much!

First of all your plugin looks awesome.
I tried to run it, opened the game map, run it and when I tried to give a command, nothing happened.
My mic works great so I guess that’s not the problem.
Probably I’m not setting it up correctly so I wanted to ask what I am doing wrong and how I can make it work.
I’m really interested in seeing how those voice commands work.

Thank you a lot.