# Speech Recognition Plugin - Sphinx-UE4

**URL:** <https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375>\
**Category:** Plugins\
**Tags:** unreal-engine\
**Created:** [December 18, 2015, 3:37pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375 "2015-12-18T15:37:30Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [January 19, 2016, 3:21am UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/21 "2016-01-19T03:21:41Z")

</div>

Hi djego,  
I believe we should keep the discussion of your issue within the github ticket you have raised.

> <https://github.com/shanecolb/sphinx-ue4/issues/2>
>
> Hi, sorry for my english, I need to help me, I have your example TestSpeech proj…ect, but I can't be work correctly.
> 
> I put "I" and inmediatly apper Message: FAILURE
> 
> I think than my microphone should be configure with UE4 but I don't know like do these, or ¿Do you have another idea?

Regards.  
Shane

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 10, 2016, 11:34pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/22 "2016-02-10T23:34:26Z")

</div>

@ - Looking into the phoneme recognition again, it _seems_ like all that really needs to be done to enable it is to pass the correct parameters and model to ps\_init and then add a new event for returning the phoneme strings - does that seem right?

From what I can tell, almost all the other code is the same as in continuous.c in the pocketsphinx source (which does phoneme recognition if the right options are passed, namely -allphone \<model file\>).

---

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [February 13, 2016, 1:25pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/23 "2016-02-13T13:25:37Z")

</div>

Pretty much man, the WordSpoken event returns the phoneme string, so no need for a new event imo.  
Let me know how it works for you. The phoneme strings where pretty abnormal from my very limited testing.

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 13, 2016, 4:41pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/24 "2016-02-13T16:41:02Z")

</div>

I did try the example program from the pocketsphinx source and notice that it spit out extra phonemes for no good reason even on its own test files. That said, I can get away with a bit more due to the resolution of current VR hardware, in that NPC and player character mouths should be small enough unless the player is right up in someone’s grill that weirdness won’t be as obvious.

Anyway I’ll see how it goes and report back!

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 13, 2016, 10:52pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/25 "2016-02-13T22:52:23Z")

</div>

Alright, well, I tried it and it does indeed spit out phonemes. Unfortunately it seems pretty random what it gives you, and it’s also a bit slow.

This is me saying “hello”. The “SIL” is just denoting silence. I couldn’t really get it to reliably spit out the right sounds, and it seems to often have a very large (2-3 seconds or more) delay before outputting the results.

```auto

LogBlueprintUserMessages: [MyCharacter2_C_2] Speech Recognition Init : SUCCESS
SpeechRecognitionPlugin: SIL G OW L OW L HH SIL M AH L

```

I’m going to play around with using some other models and see if I can get it to be a bit more reliable.

Makes me wonder what Oculus are using for the recent OVRLipSync plugin for Unity.

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 14, 2016, 12:23am UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/26 "2016-02-14T00:23:13Z")

</div>

Update: Getting faster detection using VoxForge’s dictionary and language model. I need to see about outputting a phonetic LM based on those, though.

Edit:

So not passing in anything after -allphone actually gives _much_ better results, for some reason. Not sure what’s going on, but I did manage to get this output from me saying “Test”:

```auto

SpeechRecognitionPlugin: SIL T EH S T Z F SIL

```

So the above is just with the default models recommended on the wiki, and I’ve made one change to the plugin, which is to simply pass the “-allphone” parameter with no following file parameter.

It’s also quite fast 🙂

Edit 2:

Set up an example using one of my Fuse characters by roughly translating the visemes to the values for the morphs needed to make that shape of the mouth, and set it up to split the phonemes, then blend between them for the whole spoken phrase.

It would be a lot better to have it end the utterance earlier and spit out phonemes while’s it’s still going, so that you don’t have to stop talking to start seeing the morphs go. I’m not exactly sure how to set that up.

Anyway, on the whole it definitely works and is decent enough for rough lipsync, especially good IMO for doing multiplayer voip!

I’ll be getting a video soon, though I’m going to have to shift the audio a bit to make it not look terrible 😛

Edit 3: By commenting out the bit that waits for silence before trying to figure out a hypothesis, I’ve made it work in sync with your voice.

I tried to set it up so it would also pass out the frame times for each phoneme, but there’s a bug with custom events and arrays that’s fixed in 4.11p2 onward, so I’ll have to wait for that or wrap it in a struct.

---

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [February 14, 2016, 1:12pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/27 "2016-02-14T13:12:58Z")

</div>

Awesome mate, would love to see the code and video.  
Able to pm with a link to both sometime? 🙂 cheers

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 14, 2016, 6:49pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/28 "2016-02-14T18:49:48Z")

</div>

Absolutely - I’ll be taking a new vid (the ones I took last night aren’t good enough!) and will mention the code changes in detail a bit later today.

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 14, 2016, 7:26pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/29 "2016-02-14T19:26:29Z")

</div>

First, code changes, these are really simple.

Change config to look like this

```auto

	// Start Sphinx
	config = cmd_ln_init(NULL, ps_args(), 1,
		"-hmm", modelPath.c_str(),
		"-dict", dictionaryPath.c_str(),
		"-allphone", "",
		"-bestpath", "yes",
		NULL);

```

Passing “-allphone” with a blank file parameter.

Next, I had to comment out the part where it sets phrases to recognize, as it will simply keep doing that if these are set:

```auto

	//ps_set_keyphrase(ps, "keyphrase_search", const_copy, tollerences, phraseCnt);
	//ps_set_search(ps, "keyphrase_search");

```

And lastly, and this gives you the “real time” results, commenting out the bit that makes it wait for silence before forming a hypothesis:

```auto

if (/*!in_speech && */ utt_started) {

```

So if utt\_started is true at all, we start spitting out results.

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [February 14, 2016, 9:32pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/30 "2016-02-14T21:32:24Z")

</div>

Looks a little choppy because 1) mouth shapes are just my interpretations of the visemes into Fuse’s morphs, so they’re not that accurate and 2) blending between morphs in my project is currently a bit hacky and 3) it’s not taking into account how long the phonemes are, it’s just doing each the same amount of time.

[![](https://d3kjluh73b9h9o.cloudfront.net/original/4X/1/f/5/1f50cbb7177b1c95c19a53859a5231dafb616315.jpeg "Dungeon Survival - First Look @ Automatic Lip Sync") ](https://www.youtube.com/watch?v=as8P8SseZ-I)

---

<div class="post-metadata">

**Author:** ![Pr0t0Ss12](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Pr0t0Ss12](https://forums.unrealengine.com/u/Pr0t0Ss12)\
**Post date:** [March 31, 2016, 4:47pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/31 "2016-03-31T16:47:27Z")

</div>

I tried your plugin and it works) but strange, i tryed 3 words: “spawn, one, two” in dictionary mode. All the time he recognizes “two”, what’s wrong? And phoneme mode doesn’t work at all.

---

<div class="post-metadata">

**Author:** ![anonymous\_user\_268a0caa](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@anonymous\_user\_268a0caa](https://forums.unrealengine.com/u/anonymous_user_268a0caa)\
**Post date:** [March 31, 2016, 4:58pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/32 "2016-03-31T16:58:59Z")

</div>

You were able to make it work? I kept getting “speech recognition not compatible”. I used the 4.11 dll…

---

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [April 1, 2016, 5:18am UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/33 "2016-04-01T05:18:56Z")

</div>

Pr0t0Ss12 , from the wiki:  
**"At the moment, this plugin should be used to detect phrases. (eg. “open browser”). Singular words recognition is poor. I am looking at ways to improve this to a passable level."**  
Ignore the Phoneme mode for now.

I have noticed single numbers are especially poorly recognized.  
I suspect much of this is relates to issues with Pocketsphinx.  
For example, the word “two” in the dictionary is made of two phoneme’s.  
From =short&s]=words"]CMU Sphinx Tutorial]([Building a language model – CMUSphinx Open Source Speech Recognition](http://cmusphinx.sourceforge.net/wiki/tutoriallm?s))

“For the best accuracy it is better to have keyphrase with 3-4 syllables. Too short phrases are easily confused.”

If you feel you can make improvements to the accuracy, by all means, download the source and get working on it. 🙂  
It’s available on GitHub. [GitHub - shanecolb/sphinx-ue4: A speech recognition plugin for Unreal Engine 4. This is essentially a](https://github.com/shanecolb/sphinx-ue4)  
Please let me know if you make some significant progress.

thewolfgoddess, the 4.11 binaries were built against the preview releases of 4.11.  
I see now that the main release build is available, so it’s possible the previous binaries are not compatible.  
Later tonight, I`ll download the main release build of 4.11, and investigate.

---

<div class="post-metadata">

**Author:** ![anonymous\_user\_268a0caa](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@anonymous\_user\_268a0caa](https://forums.unrealengine.com/u/anonymous_user_268a0caa)\
**Post date:** [April 1, 2016, 6:37pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/34 "2016-04-01T18:37:19Z")

</div>

Ah, alright that makes sense. I rolled it back to 4.10 and got it working. I can try to work on the accuracy if I am able, but at the moment I am trying to finish a project before the deadline XD I have noticed that it works better with phrases, or it goes a little wonky, but I am now having a different issue… Is this plugin only compatible with Speech to text? I have been trying to get it to trigger actions and events but it only seems to work when using it to output strings. Otherwise it just does… nothing. I am probably doing something wrong lol

---

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [April 2, 2016, 4:05pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/35 "2016-04-02T16:05:17Z")

</div>

I have added a section to the wiki:

[https://wiki.unrealengine.com/Speech\_Recognition\_Plugin#Sphinx-UE4](https://wiki.unrealengine.com/Speech_Recognition_Plugin#Sphinx-UE4)

> [@](#):
>
> Example Projects
> 
> The following are links to example projects, which showcase how the plugin can be used to control a character. This functionality is showcased in the following video:  
> [https://youtube.com/watch?v=wtXIiFTre\_w](https://youtube.com/watch?v=wtXIiFTre_w)  
> UE4.10 [https://drive.google.com/open?id=0BxR5qe2wdwSLWUNBSWZhNGFFSXc](https://drive.google.com/open?id=0BxR5qe2wdwSLWUNBSWZhNGFFSXc)
> 
> UE4.11 [https://drive.google.com/open?id=0BxR5qe2wdwSLWk5HR1hpU1o0LVU](https://drive.google.com/open?id=0BxR5qe2wdwSLWk5HR1hpU1o0LVU)

Please take a look. 🙂 and let me know of any problems.  
I have created an experimental branch in Github. (this is what is used by the above demo)  
For now, the major changes are correcting the enum spelling + more granular tolerance settings.  
However, in the future, I plan to change this up considerably, and merge back when it is significantly improved in all respects.

Tomorrow, I plan to create some C++ examples, that will show-case the plugin as well.

---

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [April 15, 2016, 10:58am UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/36 "2016-04-15T10:58:36Z")

</div>

A number of people had asked how to use my plugin within C++, instead of just blueprints.  
I have updated the code (experimental branch), as well as updated the example projects to include an example.  
Please take a look, and let me know if there’s any serious issues

> [@](#):
>
> Example Projects
> 
> The following are links to example projects, which showcase how the plugin can be used to control a character. This functionality is showcased in the following video:  
> [https://youtube.com/watch?v=wtXIiFTre\_w](https://youtube.com/watch?v=wtXIiFTre_w)  
> Within the project, there exists 2 blueprints. They are functionally identical, but showcase how the plugin can be used in Blueprint Only, or through C++.
> 
> SpeechRecognitionCharacter\_C\_bp: This blueprint is coded entirely in the parent C++ class. The blueprint simply exists for me to easily setup a Mesh/Animation.
> 
> SpeechRecognitionCharacter\_bp: This blueprint is coded entirely in Blueprint.
> 
> UE4.10 [https://drive.google.com/open?id=0BxR5qe2wdwSLUTBnMHlOb0hVSHc](https://drive.google.com/open?id=0BxR5qe2wdwSLUTBnMHlOb0hVSHc)
> 
> UE4.11 [https://drive.google.com/open?id=0BxR5qe2wdwSLYWc4YTI0cF9WNzg](https://drive.google.com/open?id=0BxR5qe2wdwSLYWc4YTI0cF9WNzg)

My HTC Vive arrived about a week ago, otherwise the Android port will probably have been done by now.  
Hopefully I can focus on that this weekend.

---

<div class="post-metadata">

**Author:** ![motorsep](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/motorsep/32/918698_2.png) [@motorsep](https://forums.unrealengine.com/u/motorsep)\
**Post date:** [April 15, 2016, 3:20pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/37 "2016-04-15T15:20:56Z")

</div>

Yay for Android port! 🙂

---

<div class="post-metadata">

**Author:** ![n00854180t](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@n00854180t](https://forums.unrealengine.com/u/n00854180t)\
**Post date:** [April 15, 2016, 4:46pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/38 "2016-04-15T16:46:45Z")

</div>

So I’m not sure why I had some good results with the phoneme recognition and then it later went to crud, but I’m going to start testing this again soon and see if I can get you a version that gives the results I saw before.

---

<div class="post-metadata">

**Author:** ![ShaneC](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@ShaneC](https://forums.unrealengine.com/u/ShaneC)\
**Post date:** [April 15, 2016, 5:59pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/39 "2016-04-15T17:59:54Z")

</div>

No problem. I think I recall reading that their phoneme recognition wasn’t great 😕 by their own admission.  
Let me know if you make any headway.  
I`m going to look at adding a Grammar support, and have it switching based on trigger-able contexts and such.

---

<div class="post-metadata">

**Author:** ![motorsep](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/motorsep/32/918698_2.png) [@motorsep](https://forums.unrealengine.com/u/motorsep)\
**Post date:** [April 15, 2016, 6:18pm UTC](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375/40 "2016-04-15T18:18:42Z")

</div>

Would it be better (and hopefully easy) to utilize Android’s native speech recognition on Android ?

[Previous page](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375.md?page=1)

[Next page](https://forums.unrealengine.com/t/speech-recognition-plugin-sphinx-ue4/50375.md?page=3)