[FREE] AzSpeech plugin: Text-to-Speech, Speech-to-Text and more with Microsoft Azure Cognitive Services

Update topic description:

image

Created this plugin for a personal project and I’ve decided to share with the community for free. : )


A plugin integrating Azure Speech Cognitive Services to Unreal Engine with simple functions which can do these asynchronous tasks:

  1. Voice-To-Text (Convert a speech into a string)
  2. Text-To-Voice (Convert a string into a speech)
  3. Text-To-Wav (Convert a string into a .wav audio file)
  4. Wav-To-Text (Convert a .wav audio file into a string)
  5. Text-To-Stream (Convert a string into a audio data stream)

And helper functions:

  • Runtime USoundWave importer via Audio File
  • Runtime USoundWave importer via Audio Data Stream

Links:
Marketplace: UE Marketplace
GitHub: lucoiso/UEAzSpeech
Documentation: UEAzSpeech/README.md
Microsoft Documentation: Speech Service - Microsoft Documentations

Hey, just wanted to start off by saying thanks so much for the plugin, it’s been so helpful in my work.
There is an issue I have come across where I’ve been combining the ‘Text to Stream Async’ and ‘Convert Stream To Sound Wave’ blueprint to create a sound wave from an input string and the blueprints work correctly the first time, but following string inputs always output the first sound wave despite taking a new string input.


Is this a bug or am I missing something?

1 Like

Thanks for your feedback! : )

Oops, it’s a bug. I’ll provide a update fixing it soon.

2 Likes

Now you can support me and my projects via GitHub sponsor! : )

image

Link: Sponsor @lucoiso on GitHub Sponsors :heart:

1 Like

Love this plug in, been using it for a couple different things already.

One thing I can’t figure out though is if there’s a way using its Text to Wav function to call for the voice subtypes? Like “Jenny” has Angry, Shouting, Terrified, etc. Is there a way to call those? I tried using the markup on their sample page but it just reads it out.

1 Like

Hi, i am using your plugin for a project and it is working great for one language, but i’m struggling because the speaker may use english or portuguese. Speech-to-text recognition with at-start language identification is supported by Azure, but I don’t know how to implement this. Could you help me?

Reference:

1 Like

Definitely would be great if it would have the option to support SSML (speaking styles) or maybe import a list of available voices for Azure Text To Speech?

Still, thank you for the nice plugin.

1 Like

Hey,

Great plugin, works nicely on Windows.

Question: will this support Android (Oculus Quest Developer here)? Ik would be nice to build a voice command system with this as base !

more specifically, this is what I get when trying to build for Android:


    ld.lld: error: undefined symbol: speech_config_from_subscription
    >>> referenced by speechapi_cxx_speech_config.h:47 (D:/build/U5M-Marketplace/Sync/LocalBuilds/PluginTemp/HostProject/Plugins/AzSpeech/Source/ThirdParty/AzureWrapper/include\speechapi_cxx_speech_config.h:47)
    >>>               C:/GBW_MyPrograms/Epic Games/UE_4.27/Engine/Plugins/Marketplace/AzSpeech/Intermediate/Build/Android/UE4/Development/AzSpeech/Module.AzSpeech.cppa8.o:(Microsoft::CognitiveServices::Speech::SpeechConfig::FromSubscription(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&))

    ld.lld: error: undefined symbol: speech_config_get_property_bag
    >>> referenced by speechapi_cxx_speech_config.h:463 (D:/build/U5M-Marketplace/Sync/LocalBuilds/PluginTemp/HostProject/Plugins/AzSpeech/Source/ThirdParty/AzureWrapper/include\speechapi_cxx_speech_config.h:463)
    >>>               C:/GBW_MyPrograms/Epic Games/UE_4.27/Engine/Plugins/Marketplace/AzSpeech/Intermediate/Build/Android/UE4/Development/AzSpeech/Module.AzSpeech.cppa8.o:(Microsoft::CognitiveServices::Speech::SpeechConfig::FromSubscription(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&))

    ld.lld: error: undefined symbol: property_bag_set_string
    >>> referenced by speechapi_cxx_speech_config.h:188 (D:/build/U5M-Marketplace/Sync/LocalBuilds/PluginTemp/HostProject/Plugins/AzSpeech/Source/ThirdParty/AzureWrapper/include\speechapi_cxx_speech_config.h:188)
    >>>               C:/GBW_MyPrograms/Epic Games/UE_4.27/Engine/Plugins/Marketplace/AzSpeech/Intermediate/Build/Android/UE4/Development/AzSpeech/Module.AzSpeech.cppa8.o:(Microsoft::CognitiveServices::Speech::SpeechConfig::SetSpeechSynthesisLanguage(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&))
    >>> referenced by speechapi_cxx_speech_config.h:208 (D:/build/U5M-Marketplace/Sync/LocalBuilds/PluginTemp/HostProject/Plugins/AzSpeech/Source/ThirdParty/AzureWrapper/include\speechapi_cxx_speech_config.h:208)
    >>>               C:/GBW_MyPrograms/Epic Games/UE_4.27/Engine/Plugins/Marketplace/AzSpeech/Intermediate/Build/Android/UE4/Development/AzSpeech/Module.AzSpeech.cppa8.o:(Microsoft::CognitiveServices::Speech::SpeechConfig::SetSpeechSynthesisVoiceName(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&))
    >>> referenced by speechapi_cxx_speech_config.h:168 (D:/build/U5M-Marketplace/Sync/LocalBuilds/PluginTemp/HostProject/Plugins/AzSpeech/Source/ThirdParty/AzureWrapper/include\speechapi_cxx_speech_config.h:168)
    >>>               C:/GBW_MyPrograms/Epic Games/UE_4.27/Engine/Plugins/Marketplace/AzSpeech/Intermediate/Build/Android/UE4/Development/AzSpeech/Module.AzSpeech.cppa8.o:(Microsoft::CognitiveServices::Speech::SpeechConfig::SetSpeechRecognitionLanguage(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&))


<a whole bunch of other undefined symbols>

    ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
    clang++: error: linker command failed with exit code 1 (use -v to see invocation)
Took 22,2814802s to run UnrealBuildTool.exe, ExitCode=6
UnrealBuildTool failed. See log for more details. 
AutomationTool exiting with ExitCode=6 (6)
BUILD FAILED


1 Like

SSML (Speaking Styles) would be a great addition! Thanks for your suggestions. @Soulast @Sletchman2
Hi @4TC0, i’ll take a look at this feature and give a return but it seems to be a very good feature!.
Hello @BioMedKUL, sorry about this, i’ll take a look at this issue and will try to resolve this.


I’ve spent the last few days focusing on graduating and getting a new job, but I’m coming back with development, and i’ll be adjusting Android compatibility and implementing new features based on your suggestions! : )

2 Likes

Update about Android Support:

Issue on GitHub:
Packaging errors in Android projects. · Issue #5 · lucoiso/UEAzSpeech · GitHub

Last Comment:

Tested the last commit:

  1. Created a project with a empty level and a blueprint widget that have only a button that calls Text-To-Voice when clicked
  2. Packaged the project for Android arm64-v8a: Successfull
  3. Installed and opened the project in a Xiaomi Note 8: Sucessfull

Next steps:
Will implement other buttons inside the test widget on the test project to validate all functions.

2 Likes

Closed 3 issues with the last update: :partying_face:

Pull request: bug/AND-PACK-5 and other issues: #3, #5 & #6 (github.com)
Issues resolved (if you encounter any bug, please let me know):

  1. Packaging errors in Android projects. (github.com)
  2. Wave File related functions not working on Android packaged builds. (github.com)
  3. Convert Stream to Sound Wave doesn’t work in a packaged game (github.com)

Tested with a packaged Android build in a Xiaomi Note 8:

2 Likes

The last update fixes Android compatibility and this is a great thing since there’s so many great mobile devs! : )

About other platforms:
I don’t have a MacOS, iOS device and neither a Linux device to test, but I’ve created some issues and experimental branches on GitHub if you’re interested to test it and try by yourself:

  1. MacOS:
    a) Issue: MacOS support (github.com)
    b) Branch: feature/MAC-SUP-9 (github.com)

  2. iOS:
    (Currently incomplete - need to search about iOS UPL and iOS permissions)
    a) Issue: iOS support (github.com)
    b) Branch: feature/IOS-SUP-10 (github.com)

  3. Linux:
    a) Issue: Linux support (github.com)
    b) Branch: feature/LINUX-SUP-8 (github.com)


Also added 2 new issues based on suggestions:

  1. SSML: Improve the plugin by adding SSML support (github.com)
  2. Language Identification: Improve the plugin by adding Language Identification support (github.com)

The plugin is open-source, you are free to fork the repository and modify everything you want. : )
And feel free to send a pull request and contact me if you’re interested to contribute with the project.

2 Likes

Hi UE 5 user here, @lucoiso today installed the plugin, and not using it yet, but when packaging getting this error:

s): libpugixml-windows.lib(pugixml.obj) : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance
UATHelper: Packaging (Windows): Creating library D:\UNREAL\CS\Binaries\Win64\ClimateState.lib and object D:\UNREAL\CS\Binaries\Win64\ClimateState.exp
UATHelper: Packaging (Windows): Module.AzSpeech.cpp.obj : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize
UATHelper: Packaging (Windows): Module.AzSpeech.cpp.obj : error LNK2001: unresolved external symbol __imp___std_init_once_complete
UATHelper: Packaging (Windows): D:\UNREAL\CS\Binaries\Win64\ClimateState.exe : fatal error LNK1120: 2 unresolved externals

Also this error (low priority)

LogBlueprint: Error: ProxyFactoryClass null in K2Node_AsyncAction /Game/System/UI/UI_Main.UI_Main:EventGraph.K2Node_AsyncAction_0. Was a class deleted or saved on a non promoted build?

Notice, there was a node placed but not connected to anything. Unclear if this is new error with UE 5 or also due to some settings of your plugin. Bottom line have to remove plugin and unconnected nodes to build.

1 Like

Hi! @unit23 : )

Have you tried verifying the engine in Epic Games Launcher? And can you package without errors with the plugin disabled?

And are you using which OS? This plugin is currently only working on Win64 (there’s other experimental branches on github but not tested)

I’m asking this because I can’t replicate the error here.
Tested packaging for Android and Windows using a Windows 11 PC and both completed with success. :eyes:

1 Like

I use 5.0.2 with Windows 10, and can package without issues when your plugin is disabled.

1 Like

Hey @unit23! : )

Did you download through Unreal Marketplace or GitHub versions?
If you are using the Marketplace version, can you test with the GitHub version?

Download: AzSpeech v3.0.7 (github.com)

I found a similar problem here:
This plugin (Microsoft OpenXR) isn’t working properly if downloaded via Marketplace because “This is a regression in how the Unreal Marketplace is packaging HoloLens plugins. They are investigating.”

So i decided to try replicating the problem again using the marketplace version (i was using the source code inside a test project) but now i got the same errors.

For some reason, the version that Epic sent to marketplace is not working as expected, I will try to contact them about this and look for a solution.

Hope to solve this soon. : S


The problem is solved and was related to the privacy of some dependency modules: Marketplace versions are returning errors when trying to package a game with this plugin · Issue #13 · lucoiso/UEAzSpeech (github.com) :partying_face:

1 Like

AzSpeech v4.0.0

On GitHub: GitHub Release
On Marketplace: Unreal Marketplace

What’s Changed

Full Changelog: Comparing v3.0.7...v4.0.0 · lucoiso/UEAzSpeech · GitHub

Hi, I’m getting the following message when trying to build to my Oculus Quest using UE4.27.
Could be a noob question but I’m not sure how to fix this.
Building to Android (Quest 2) does work fine in this project, except when the plugin is enabled, I get this error. I’ve also manually added the .h file in the root but then I get the message that there’s a duplicate file which seems to indicate that the system does find the file but for some reason it still gives an error.

If you have any suggestions, that would be great :slight_smile:

1 Like

Hi @BioMedKUL !

This is because the dependency (AndroidPermission) is declared as private on the current version of the AzSpeech on marketplace. :S

For some reason, when using the plugin inside the project folder you can package without problem, but using the marketplace version returns some errors.

The latest version solved this by declaring AndroidPermission (and AzureWrapper) as a public module dependency inside the build.cs file and declaring the AndroidPermission plugin as dependency (as optional because this is only needed for Android) on the .uplugin file.


Maybe you’ll need to enable the plugin AndroidPermission too. Because the plugin needs permission to access user’s microphone to do some functions such as Voice-toText.

1 Like

Hello!

I would like to report an issue that happens with Text to WAV function. When using a filepath that does not exists, it tries to create the filepath and save the WAV inside it, and this is working only when the choosen filepath is inside an existing folder, but it does not work when the filepath is a subfolder of a folder that does not exists.

Exemple of an empty folder:
“C:\UnrealEngine”

If I try to save something like “C:\UnrealEngine\TextToWav\test” (“test” being a subfolder of “TextToWav”, both don’t exist), it will not create “TextToWav” folder, then create “test” inside it. It will not work.

But if I try to save with filepath “C:\UnrealEngine\TextToWav” (just a file that does not exists), it will create the “TextToWav” folder and save the WAV inside it.

Do you know if this problem can be solved? Thanks!

1 Like