[FREE] AzSpeech plugin: Text-to-Speech, Speech-to-Text and more with Microsoft Azure Cognitive Services

Hi,

I updated to latest version today, and now I started getting crashes after one voice recording in editor. It only triggers if I do something after with the voice input. When it’s not a command I recognise (like the first example in logs), it’s ok, but when I recognize the command and try to do something afterwards, it crashes.
The project settings are still ok.
When I build to device (Quest 2), the same happens, it records 1 voice and crashes afterwards.
It’s visible in code → if I go through my branches and hit a dead end, everything is fine, if I change that dead end to a print (or go any of the other true gates), the crash occurs.

The logs:
[2022.09.21-14.11.58:032][864]LogAzSpeech: Display: AzSpeech - AsyncVoiceToText: Initializing task
[2022.09.21-14.11.58:046][865]LogAzSpeech: Display: AzSpeech - GetAzureRecognizer: Using language: nl-BE
[2022.09.21-14.12.04:080][224]LogAzSpeech: Display: AzSpeech - ProcessAzSpeechResult: Task completed. Reason: RecognizedSpeech
[2022.09.21-14.12.04:197][231]LogBlueprintUserMessages: [BP_Medic_2] Breng me de aivd.
[2022.09.21-14.12.04:197][231]LogAzSpeech: Display: AzSpeech - AsyncVoiceToText: Result: Breng me de aivd.
[2022.09.21-14.12.05:619][316]LogBlueprintUserMessages: [BP_Medic_2] looked at medic
[2022.09.21-14.12.05:619][316]LogAzSpeech: Display: AzSpeech - AsyncVoiceToText: Initializing task
[2022.09.21-14.12.05:636][317]LogAzSpeech: Display: AzSpeech - GetAzureRecognizer: Using language: nl-BE
[2022.09.21-14.12.10:175][586]LogAzSpeech: Display: AzSpeech - ProcessAzSpeechResult: Task completed. Reason: RecognizedSpeech
[2022.09.21-14.12.10:247][590]LogBlueprintUserMessages: [BP_Medic_2] Breng de AED.
[2022.09.21-14.12.10:247][590]LogOutputDevice: Warning:

Script Stack (2 frames):
BP_Medic_C.ExecuteUbergraph_BP_Medic
BP_Medic_C.TaskCompleted_8BF2DFC04836D8FBA6242187C6B14664

[2022.09.21-14.12.10:247][590]LogWindows: Windows GetLastError: The operation completed successfully. (0)
[2022.09.21-14.12.11:099][612]LogThreadingWindows: Error: Runnable thread TaskGraphThreadBP 28 crashed.
[2022.09.21-14.12.11:100][612]LogWindows: Error: === Critical error: ===
[2022.09.21-14.12.11:100][612]LogWindows: Error:
[2022.09.21-14.12.11:100][612]LogWindows: Error: Assertion failed: !bPostTickComponentUpdate [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/Engine/Private/LevelTick.cpp] [Line: 897]
[2022.09.21-14.12.11:100][612]LogWindows: Error:
[2022.09.21-14.12.11:100][612]LogWindows: Error:
[2022.09.21-14.12.11:100][612]LogWindows: Error:
[2022.09.21-14.12.11:100][612]LogWindows: Error: [Callstack] 0x00007ff8dd674fd9 KERNELBASE.dll!UnknownFunction []
[2022.09.21-14.12.11:100][612]LogWindows: Error: [Callstack] 0x00007ff8a48739b8 UE4Editor-Core.dll!UnknownFunction []
[2022.09.21-14.12.11:100][612]LogWindows: Error: [Callstack] 0x0000008fedef9ee0 UnknownFunction []
[2022.09.21-14.12.11:100][612]LogWindows: Error: [Callstack] 0x0000000000000400 UnknownFunction []
[2022.09.21-14.12.11:100][612]LogWindows: Error: [Callstack] 0x0000000000000068 UnknownFunction []
[2022.09.21-14.12.11:100][612]LogWindows: Error: [Callstack] 0x0000000000004000 UnknownFunction []
[2022.09.21-14.12.11:100][612]LogWindows: Error:
[2022.09.21-14.12.11:100][612]LogWindows: Error: Crash in runnable thread TaskGraphThreadBP 28
[2022.09.21-14.12.11:122][613]LogExit: Executing StaticShutdownAfterError
[2022.09.21-14.12.11:122][613]LogWindows: FPlatformMisc::RequestExit(1)
[2022.09.21-14.12.11:122][613]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)
[2022.09.21-14.12.11:122][613]LogCore: Engine exit requested (reason: Win RequestExit)
[2022.09.21-14.12.11:128][613]Log file closed, 09/21/22 16:12:11

1 Like

Hi! : )

Answered here: Experimental: Support for MacOS Platform · Issue #9 · lucoiso/UEAzSpeech · GitHub

Hi @HOTOREL !

Unfortunately I don’t have iOS and Mac devices to test, so it’s hard to solve this, sorry :confused:

That’s why I left it as experimental, too bad it didn’t work. :frowning:


If you want to try, there are some files where some implementations are made:

To work in the editor, it is necessary to load the libraries (I probably implemented it wrongly for MacOS/iOS in the branch feature/MAC-SUP-9, but as I have no way of testing it here, so it is difficult to adjust): https://github.com/lucoiso/UEAzSpeech/blob/main/Source/ThirdParty/AzureWrapper/AzureWrapper.Build.cs & https://github.com/lucoiso/UEAzSpeech/blob/main/Source/AzSpeech/Private/AzureSpeech.cpp

For android it was necessary to have an .xml file (UPL - Unreal Plugin Language): https://github.com/lucoiso/UEAzSpeech/blob/main/Source/ThirdParty/AzureWrapper/AzSpeech_UPL_Android.xml

Hi! : )

Can you check if this is occurring in this version? AzSpeech v5.0.4 Latest yesterday (github.com)

Note: This change will be on the marketplace soon


Made some tweaks to the output broadcast, reversing a change from a previous commit and setting the delegate to call only on GameThread using a copied value to prevent the result from being dereferenced or other undefined behavior


Sorry for the delay in responding, I was having a problem related to publishing updates on the marketplace and other jobs are taking up a lot of my time lately

Hi!

Just tried it out, and indeed it’s working now, thanks a lot!

No problem about the delay, we are all human :wink: It is already so great you are providing this plugin and the excellent support!

On a related note, we had a question about background noise, atm the service also picks up voice from other people in the office, do you know of any way to put filtering of that? We’ve thought about using the ‘set microphone threshold level’ node, but that won’t cover most of our use cases, and we’re not really familiar with noise suppression of filtering.

If there’s not really a straight answer, we have time to research it, so don’t feel pressured!

1 Like

:white_check_mark: Working with the current UE5.1 Preview 1 ver.

Branch: lucoiso/UEAzSpeech at UE5.1 (github.com)


Changes: Created a new branch and removed the .uplugin param “EngineVersion”

1 Like

Hey :slight_smile: after the latest windows update “2022-10 Cumulative Update for Windows 11 for x64-based Systems (KB5018418)” the plugin stopped working, it will not generate anything, and all tasks show up as timed out. Have you seen this issue? Do you know what might fix it? Thanks a lot :slight_smile:

2 Likes

Same issue I am facing, Please let me know if any one knows the solution.
Thanks in Advance.

1 Like

There’s a bug with the Microsoft’s SDK and they’re investigating this issue: Text to speech timeouts with error: WS_OPEN_ERROR_UNDERLYING_IO_OPEN_FAILED / USP error: timeout waiting for the first audio chunk · Issue #1692 · Azure-Samples/cognitive-services-speech-sdk (github.com)

I also created a new issue (and a branch with better logs): Synthesizers stopped working after Windows KB5018418 update · Issue #29 · lucoiso/UEAzSpeech (github.com)

1 Like

I’ve got exatly same plroblem, SSML to Audio Stream, or Voice, or WAV on Windows 10 pro, ver 21H2 19044, 2130.
Tho, if I spam the request it goes through sometimes. Like once per twenty requests.

@AdamEVEGames @kkunreal1 @Z-enzyme

Take a look at this GitHub Issue. Microsoft released some fixes! :eyes:

Win 10: October 17, 2022—KB5020435 (OS Builds 19042.2132, 19043.2132, and 19044.2132) Out-of-band (microsoft.com)

Win 11: October 17, 2022—KB5020387 (OS Build 22000.1100) Out-of-band (microsoft.com)

Other ver.: Text to speech timeouts with error: WS_OPEN_ERROR_UNDERLYING_IO_OPEN_FAILED / USP error: timeout waiting for the first audio chunk · Issue #1692 · Azure-Samples/cognitive-services-speech-sdk · GitHub

I geniuenly have no idea what to do with this information. And I really need it for work.
Tried to instal KB5020436 from windows update catalog but there are only server versions, and they don’t work.

Edit: oops, tried to install the wrong update. works like a charm now.
For anyone struggling, on Win 10, go to Microsoft Update Catalogue, search for KB5020435, download, install, profit.

1 Like

To help identify future issues, I’m enhancing logging, adding tests and adding some new features.

I’m currently working in this branch: lucoiso/UEAzSpeech at feature/STOP-SYNTH-28 (github.com)

Current changes:

  1. Updated Azure Speech SDK
  2. Added better logs
  3. Enabled SDK logs (generated inside Saved/Logs/AzSpeech when a task fails)
  4. Added the possibility to stop synthesizers tasks
  5. Created tests: UEAzSpeech_Tests (github.com)

Examples

  1. Stop Synthesizer Task:
    image

  2. SDK Log:
    UEAzSpeech 2022.10.20-18.02.24.log (25.6 KB)

2 Likes

The plugin works great; however zh-CN and zh-tw isnt working

, please help me , thanks

1 Like

An update about the synthesizers failing:

Microsoft published a update for 22H2 versions: Microsoft Update Catalog


Azure SDK - Github Issue Link

Hi! @anonymous_user_6726f6a7 : )

That’s strange… zh-CN is a supported language: Language support :eyes:

Are you getting errors in the log?

And currently there’s a known issue occurring with some people after update the Windows.

If you’re facing the same problem (using Microsoft Windows), Microsoft has released some updates to fix this:

Windows 10: October 17, 2022—KB5020435 (OS Builds 19042.2132, 19043.2132, and 19044.2132) Out-of-band (microsoft.com) 2

Windows 11 21H2: October 17, 2022—KB5020387 (OS Build 22000.1100) Out-of-band (microsoft.com) 1

Windows 11 22H2: Microsoft Update Catalog


If this issue is not applicable to your situation, please let me know


There’s a branch on github (with some implementations of the next update) with better logs and with the Azure SDK log enabled: lucoiso/UEAzSpeech at feature/STOP-SYNTH-28 (github.com)

There’s a .zip of the current state of this branch (is not the final version, errors may occur):
AzSpeech.zip (4.3 MB)

This will give you access to some of the changes I’ve made, including the opportunity to stop some tasks, and you’ll have access to your Azure logs in case you encounter any errors.

1 Like

AzSpeech v1.1.0

Release: AzSpeech v5.1.0 (github.com)
Marketplace: Waiting for approval

What’s Changed

Pull Request: v1.1.0 by @lucoiso in v5.1.0 by lucoiso · Pull Request #32 · lucoiso/UEAzSpeech · GitHub

Full Changelog: https://github.com/lucoiso/UEAzSpeech/compare/v5.0.4...v5.1.0

Hi, Thank you for the great plugin.
Could you please tell , Is the VoiceToText will work in Pixel Streaming ?

1 Like

I don’t know the answer now, I’ll do some tests to be sure! And if it doesn’t work, I’ll look for a way to add compatibility! : )

Some upcoming features

I’m currently working on viseme and continuous recognition support! : )

Branch: feature/VISEME-22 (github.com)

Issues:

  1. Viseme support (github.com)
  2. Add support to continuous recognition (github.com)

Support for Viseme:

Log Output Example:

[…]
[2022.11.06-19.06.49:180][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Id: 6
[2022.11.06-19.06.49:180][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Audio Offset: 33500000
[2022.11.06-19.06.49:180][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Animation:
[2022.11.06-19.06.49:180][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Id: 20
[2022.11.06-19.06.49:181][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Audio Offset: 34000000
[2022.11.06-19.06.49:181][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Animation:
[2022.11.06-19.06.49:181][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Id: 19
[2022.11.06-19.06.49:181][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Audio Offset: 34500000
[2022.11.06-19.06.49:181][156]LogAzSpeech: Display: OnVisemeReceived - Viseme Animation:
[…]


Support for Continuous Recognition

2 Likes

Hey there, I am attempting to use this plugin in UE5 but I’m not having any success at all, my brother is having the exact same issue so I think we must be doing something wrong.
We are using the text to voice node, but the task completed never fires, we have found that if we put the voice name and language ID into the azspeech project settings that the task completed node will fire but it will always fail. Any help here would be amazing!
Thanks so much <3



1 Like