Download

Speech Recognition Plugin - Sphinx-UE4

Hey bud, I put up a separate thread for the OVRLipSync plugin, let’s move discussion there so as not to clutter ShaneC’s thread up XD OVRLipSync Plugin for UE4 - Community Content, Tools and Tutorials - Unreal Engine Forums

As for Android, I haven’t had time to spend messing with it, but it’s honestly just a matter of changing some defines to account for Android, and putting in the .so from the Unity package in the right folder. If you want to try your hand at it, add me on Steam (same nick as here) and I’ll walk you through it.

I know, I posted there before. I was actually asking @ShaneC :wink:

Added you, thanks, but I’m gonna have to wait until you add sound input (sound file, not mic) as I am interested in talking NPCs (single player) :o

motorsep, yes, it’s in the works. Sorry, I just haven’t had much time, and the work has been very annoying, since each testing cycle require build/deploy/test/debug on the device…much longer than Windows testing.
I have speech recognition within Android working, through an Unreal Engine environment.
It’s just the communication between the two that I am currently having some issues with.
I still think any usage will require some manual steps, and a development background. It won’t be as simple as setting it up for Windows.
I`ll have some free-time over the weekend to work on it further.

yay! Sounds like you are almost there!

I am pretty sure it’s possible to build it as .so lib, something like OVRLipSync, but either way, if you can make a short step-by-step write up how to set it up and it wouldn’t require engine rebuild, it should be fine.

Sphinx-UE4 (Version 1.0):
I have made a number of changes to the plugin. Please checkout the following link, to download the example project and test away.
Feel free to provide feedback and suggestions within the thread.

Updated the speech recognition plugin I wrote, as well as improved the demo project.

  • Fixes a bug preventing Chinese from working.
  • Compile for UE4.13 and UE4.14
  • Added a few more languages.
  • Fixed a bug where a minor stutter would occur, when initially loading the plugin.
  • Added an experimental method to obtain an approximation of the Volume
    The demo project has been expanded, I personally like how the Grammar allows a super crude calculator to exist.

Check it out :slight_smile:

Nice update!

Any news about Android support by chance ? :o

Great work! I’m interested too in the Android version of the plugin. Any update or ETA?

I think the plugin is not working once you export the project, although it is loaded as showed in the log:
“Loaded Plugin OVRLipSync”
then, it will not work (not initialized), any fix for this?

What sort of license does this have?

I have crash error when i exit project

Fatal error: [File:D:\Build++UE4+Release-4.15+Compile\Sync\Engine\Source\Runtime\Core\Private\GenericPlatform\GenericPlatformMemory.cpp] [Line: 160]
Ran out of memory allocating 42949672940 bytes with alignment 0

KERNELBASE
UE4Editor_Core
UE4Editor_Core
UE4Editor_Core
UE4Editor_Core
UE4Editor_Core
UE4Editor_SpeechRecognition!TArray<TScriptDelegate<FWeakObjectPtr>,FDefaultAllocator>::ResizeForCopy() [d:\unreal engine\epic games\ue_4.15\engine\source\runtime\core\public\containers\array.h:2281]
UE4Editor_SpeechRecognition!TMulticastScriptDelegate<FWeakObjectPtr>::TMulticastScriptDelegate<FWeakObjectPtr>()
UE4Editor_SpeechRecognition!ASpeechRecognitionActor::StartedSpeaking_method() [d:\My_Project\plugins\speechrecognition\source\speechrecognition\private\speechrecognitionactor.cpp:122]
UE4Editor_SpeechRecognition!FSpeechRecognitionWorker::Run() [d:\My_Project\plugins\speechrecognition\source\speechrecognition\private\speechrecognitionworker.cpp:488]
UE4Editor_Core
UE4Editor_Core
kernel32
ntdll

But if i not INIT Speech, exit without error.

When i press escape (to exit and close preview) i get error

Fatal error: [File:D:\Build++UE4+Release-4.15+Compile\Sync\Engine\Source\Runtime\Core\Private\GenericPlatform\GenericPlatformMemory.cpp] [Line: 160]
Ran out of memory allocating 42949672940 bytes with alignment 0

KERNELBASE
UE4Editor_Core
UE4Editor_Core
UE4Editor_Core
UE4Editor_Core
UE4Editor_Core
UE4Editor_SpeechRecognition!TArray<TScriptDelegate<FWeakObjectPtr>,FDefaultAllocator>::ResizeForCopy() [d:\unreal engine\epic games\ue_4.15\engine\source\runtime\core\public\containers\array.h:2281]
UE4Editor_SpeechRecognition!TMulticastScriptDelegate<FWeakObjectPtr>::TMulticastScriptDelegate<FWeakObjectPtr>()
UE4Editor_SpeechRecognition!ASpeechRecognitionActor::UnknownPhrase_method() [d:\my_project\plugins\speechrecognition\source\speechrecognition\private\speechrecognitionactor.cpp:106]
UE4Editor_SpeechRecognition!FSpeechRecognitionWorker::Run() [d:\my_project\plugins\speechrecognition\source\speechrecognition\private\speechrecognitionworker.cpp:561]
UE4Editor_Core
UE4Editor_Core
kernel32
ntdll

Hi, ShaneC !

could you give some instructions how to improve recognition a single phrases like “yes” or “no”, may be in cpp, please?

Hello,
Is this plugin still supported in the latest version of Unreal Engine?

Just compiled and tested on 4.16 and appears to be working for me - but I haven’t tried a packaged build yet.

Any easy way to get the text of any speech without having to match keywords or predefined phrases? Maybe get the I know this can be inaccurate, but perfect sentences is not important for what I need this for. Manager->UnknownPhrase_method(return the whatever phrase); ???

I’m having a problem with a grammar script.


#JSGF V1.0;

public <final_rule> = ( <s1> | <s2> | <s3> | <s4> | <s5> | <s6> ) <element> ( <f1> | <f2> | <f3> | <f4> | <f5> );

<s1> = <count> <shape>;
<s2> = <count> <direction>;
<s3> = <count> <direction> <shape>;
<s4> = <shape>;
<s5> = <direction>;
<s6> = <direction> <shape>;

<f1> = <range> <rate>;
<f2> = <range> <rate> <intent>;
<f3> = <range> <intent>;
<f4> = <intent>;
<f5> = <count>;

<element> = ( roth | mara | corda | ashti | kava | dolo | hana );
<shape> = ( vort | para | nama | haku | mado );
<direction> = ( chiya | kanto | vanti | bren );
<count> = ( mapa | haya );
<range> = ( juna | toda | wato );
<intent> = ( dema | cora | seto );
<rate> = ( gen | turb | rata );

Is anyone able to see what’s wrong with that? The words all exist in the dictionary and if I Enable Grammar Mode with the “numbers” script it starts fine. But if I use the script above I get “Speech recognition thread failed to start”.

Is there any way to debug this?

Edit: fixed it. I’ll share how I debugged it since it’s pretty non-obvious.

Even if you’re using grammar mode, switch to keyword mode and feed it all of your new words in a word list. Then comment all of your new words out in the .dict file using a semi-colon ;

After that, re-enable them one at a time and hit play. If it “fails to start utterance” then your syllables in the word you just uncommented are wrong. I kept getting caught out by G (should be JH), H (HH) and O (OW).

why can’t i compile this? im getting this error

UATHelper: Packaging (HTML5): In file included from E:\WLS_HELIX\AoAllowwishes\Plugins\SpeechRecognition\Source\SpeechRecognition\Public\SpeechRecognitionActor.h:4:
UATHelper: Packaging (HTML5): E:\WLS_HELIX\AoAllowwishes\Plugins\SpeechRecognition\Source\SpeechRecognition\Public/SpeechRecognitionWorker.h(3,10): fatal error: ‘sphinxbase/err.h’ file not found
UATHelper: Packaging (HTML5): #include <sphinxbase/err.h>
UATHelper: Packaging (HTML5): ^~~~~~~~~~~~~~~~~~
UATHelper: Packaging (HTML5): 1 error generated.
UATHelper: Packaging (HTML5): ERROR:root:compiler frontend failed to generate LLVM bitcode, halting

where can i get this file it is not in the folders?

This works amazingly well in 4.18! THANK YOU Wow,. I have read the entire thread before I ask you for help, but:
The once thing i cannot figure out how to do is to link an added phrase to a new Function (or Blueprint?)! I can see to add a phrase add a ‘+’ PIN to the ‘Make Array’ and I know there will be a corresponding Function. I already got it to play a Wav file as it does Start Game and nearly passed out when it worked! WOW WOW WOW…

any help greatly appreciated! THANK YOU!

EDIT: I always find things on my own right after asking, it is the ‘WordsSpoke’ Function :slight_smile: And as I hoped there is another easy to understand array!

Although I still welcome any and all advice about optimizing recognition… Phrases like thank you do not seem to work even though I found ‘thank’, ‘you’ and ‘thankyou’ in the dict file!..

THANK YOU!

same problem, have you solved it?