Microphone input for audio analysis

Hi,

I’m new to the unreal engine, and I’m trying to wrap my head around adding a feature. If it works out, it would be a nice bonus to be able to contribute something to the engine.

I’ve been looking into getting audio input for analysis, and it works great on Windows and Mac using the IVoiceCapture interface. It’s obviously intended for voice chat, but i can get at the raw data at a reasonable interval.
However, I really need this feature on iOS and android. And it seems weird to me to implement an android and iOS module under the online section of the code as the feature has nothing to do with online.

Then I considered adding it to the Media Framework, perhaps by creating a new IMediaRecorder interface. This make some sense to me, but I would perhaps have to duplicate the windows and mac module in this new recorder module.

At this point, I’m settling on creating an IMediaRecorder interface and having the online voice section call into that code. But since I’m new to the engine, this might be a terrible idea.
I’m totally happy having my local version that does this, but if someone had some opinions on how I might do this in a manner that could potentially end up as a pull request, I would feel better about myself.

Or perhaps I have completely missed that this feature is already available? That would be sweet.

Any advice is highly appreciated,

+1 for this

I added support for audio and video capture hardware in Media Framework 3.0 (currently scheduled for 4.18). It is currently only implemented for Windows (audio and video) and Android (video only), however. More platforms will be added in 4.19. You could implement your own audio capture plug-in on top of that new API though. We are planning to finally merge to Dev-Main on Thursday. If everything goes well, the new API should be on Github before the end of the week.

, do you have progress in what you are doing?

I’m able to get the data from IVoiceCapture, however it works only on windows, I cant get it on android, even the capture state is OK , I’m really confused and mad , and zero support, I couldnt get any answer about that from either answerhub or forum .

https://github.com/EpicGames/UnrealEngine/pull/2894
This was a community addition to the engine, and you must always check with the github repository when dealing with undocumented features. There could be other reasons to the capture is not functioning properly, but reading this implementation will probably lead you to get this work.

That is not entirely true, you should be able to implement any voice capture class for wide spectrum of compatibility. One that has been already done by the engine developers can be found in the engine plugins runtime section, called AudioCapture.

https://github.com/EpicGames/UnrealEngine/tree/4.19/Engine/Plugins/Runtime/AudioCapture/Source/AudioCapture/Private

While the plugin will focus on voip once again, that doesnt makes it exclusive and you should be able to use it for your purposes. It’s not dealing with handy devices, which is kind of unfortunate, however the example should be good enough to rethink your approach and find a suitable library to be implemented with the engine, similar to this plugin’s example.

I have done the rtaudio implementation separatedly, and i put the class straight in the project code, it was working great and managed to implement my voip this way (without using the online subsystem).

https://github.com/EpicGames/UnrealEngine/tree/4.17/Engine/Plugins/Editor/AudioCapture/Source/AudioCapture/Private

Here this implementation was being used for live audio capture in editor, probably a better example as it is dealing with the audio data very differently.