Karaoke - recognize speech with player microphone?

I am interested in making a Karoake game where the player sings and is judged by hitting the right note or lyric at the right time. For example, if the lyric is spoken incorrectly they fail or lose a lot of points, but if it’s close enough such as getting the main vowel right like “break” and the player says “steak”, the player would only lose a few points because they were phonetically close enough.

I know in real karaoke the words on the screen sometimes flash red when not sung incorrectly, so I know it’s not impossible, I’m just unfamiliar with that aspect of UE4.

This may be of use to you, I don’t feel comfortable enough to give a sophisticated answer on this subject matter.

Thanks for the link, I’ll check it out :slight_smile:

Note that it is more impotent to recognize tone of sound then correctness of the text and that you can do by detecting the pitch of sound. UE4 don’t provide those features out of the box so either you need to do it yourself or get plugins

Be careful not to use the Sound Visualization plugin that comes with the engine though - yes it can detect frequency and amplitude of sounds that are being output but it does not work in packaged builds. Also, it’s for output sounds; you need something that handles input sounds.

You can take a look at the RuntimeSpeechRecognizer plugin, which is based on Whisper AI. It’s cross-platform and works offline. It’s also completely free and open source.

GitHub link: GitHub - gtreshchev/RuntimeSpeechRecognizer: Cross-platform, real-time, offline speech recognition plugin for Unreal Engine. Based on Whisper OpenAI technology, whisper.cpp.

Marketplace link (But please prefer downloading from GitHub as we’ve noticed some issues with the Marketplace language model staging): Runtime Speech Recognizer in Code Plugins - UE Marketplace