Speech Recognition Plugin - Sphinx-UE4

Development Pipeline & Plugins Plugins
1

I have recently made some bug fixes, and added Chinese and French languages.

Let me know your thoughts.
Thanks :smiley:

2

@ - Awesome work man. I like that you can add words to the dictionary - that’s going to be instrumental for making a game-specific wordset. :smiley:

Not sure on the wiki issue.

Edit: Do you think you could give some pointers to where I should start in the plugin code for adding the phoneme recognition type as an option?

3

Hi n00854180t,
I have read that there are some issues with phoneme recognition.

Never the less, I am adding in some ground-work into the code :slight_smile:
This will allow a mode to be selected when initializing.
For the time being, this will be Dictionary Keyword, and Phoneme Recognition.
The modes can certainly be expanded on at a later date.
Cheers,
Shane

4

You the man!

If you need someone to test phoneme recognition I’d be happy to give it a try.

As an aside, is there currently a way to pass in a SoundWave or raw sound samples rather than using the microphone? I think phoneme recognition would be especially useful for lipsyncing voice over dialogue sounds.

5

Woah! Question: Let’s say I wanna give an AI an order using this; how would it be appliable for it to recognize the command: “go” I specify it via blueprints?

6

so i’m intersted in this plugin, can you add added Arabic languages or it’s too early for that ?

7

n00854180t,
Passing in a sound wave is certainly possible. i hadn’t gone down that path yet, as I didn’t really see a useful application (and I have limited time).
You could certainly pass in a sound wave, and have it return a time-frame of when phoneme’s were spoken.

This brings me to my next point, the approximation (from my testing) is rather approximate (or all over the shop, from my testing).
You could perhaps use this information with a phoneme dictionary list, and an advanced algorithm to match the detected/matched phoneme’s, and approximate the timing of the gaps.
I am unsure how well this would work, to be perfectly honest.

Regarding the phoneme work, I have created a branch, and will check-in my work later in the week. I don’t really have the time to thoroughly test (it seems very random)
But, this will show you how it CAN work, and should provide a starting place for your development or expansion.

,
Did you look at the blueprint example?
Simply change the Word Spoken function to perform a switch statement on the incoming text, and trigger whatever you like based on the phrase you added.
Note: Simple phrases are more error prone (eg. false positives)… especially something as simple as “Go”

Andrew, there does exist an Arabic language model for Sphinx.
I went down the path of investigation of Arabic (as another person had requested) but it seems Arabic is not supported by Unreal Engine 4.
Have you tried pasting Arabic text into an FString. It simply comes up as a series of unknown characters.

Arabic.png
Arabic.png719×277 40.6 KB

I had thought of converting the Arabic dictionary file, into a Latin transliteration, but I have no idea how that could be done on a large scale.
Ideas?

8

@ - awesome man, you’re a life saver.

I’ll definitely be checking out the phoneme branch and seeing where I can take it. I’d love to be able to create an auto-lipsync tool for UE4 using it, but we’ll see if it’s accurate/good enough for that :slight_smile:

9

Hey , I did check it, I just wasn’t so sure how well it could handle multiple transliterations. I’ll see if I can manage to create a simple AI, that obeys voice commands.

10

Hey ,
Hmm, multiple transliterations? I thought you wanted a particular voice phrase to trigger an in-game action?
The transliteration comment was for Andrew, regarding a request for Arabic speech :slight_smile:

Call Init with phrases you wish to recognise.

1.PNG

Then have a switch on the returned phrase, to trigger different logic based on the recognised phrase.

2.PNG
2.PNG1097×381 48.7 KB

You would probably also want other checks, so that only certain phrases perform certain actions at certain times, but that is really up to you.
Cheers

11

Can I run your program in MacBook UE4?

12

Hi Artigen,
At this time, the plugin does not work under Mac OS.
I would be happy to port it to Mac OS, if you were willing to pay me for my time.
Alternatively, the code is publicly available on GitHub.
All the best,
Shane

13

Has this been tested yet on 4.11?

And oh my… this is exactly the plugin I have been looking for since UE4 came out \o/

14

ddddd.png

getting this error when i am trying to open it on 4.10
i tried to build from source using vs2015.
but still same

15

They must have compiled with debugging and released that version and not the non debugging version. the d on the end signifys debugging.

16

ahh, that would be my bad. I`ll update the dll’s and let you know when I have done so.

17

Okay, I have updated GitHub, and the Wiki.

Note: There’s an additional step to copy across the appropriate dll to match the version of unreal you are using.
Please let me know if you encounter any issues.

18

sorry for the delay, about converting Arabic dictionary file into a Latin transliteration i don’t think this will work, tried it before in unity but who guess it could work in unreal :smiley:
Also i got agood news 4.11 - 4.12 will support Arabic so it can be easy to compelete your work without converting any thing :smiley:

19

Oh, that’s great news. :slight_smile: Thanks for letting me know.
I`ll let the person who e-mailed me know that Arabic is being added in 4.11-4.12.

20

Hi, i have a problem, when Run my project then I put “I” (for init Speech) my Microphone Input not work, Stop and Close project come back Work my microphone :confused:

Have you ever happen it ?