Playing realtime network audio in Unreal from custom stream source

Hello!
This is my first post in this forum, so I would like to address a sincere “thank you!” to all the people around here developing and working with the engine, its plugins and writing about it :slight_smile:

I will go to the point of my post:
in the next future, my team and I will need to play back RTP (or other) audio streams coming from a remote microphone for telepresence scenarios, within Unreal. We are capable of playing back an audio stream using ffmpeg through the ffplay tool from command line (tested on Linux and Windows).
Also, we have been using a custom C++ integration of ffmpeg within Unreal in Windows for the last 1.5 years to play real time network video streams. It is not particulary polished, but it is giving nice results in terms of performance for our purpose, so I would like to take the same route for audio.

Therefore, our plan would be that of using ffmpeg to decode RTP audio streams, and render them somehow within Unreal.
That would be a big plus for us, since at some point we might need to work on the spatialization of the audio streams.
Also, the alternative of introducing a custom audio renderer just to play those streams seems like a really counterintuitive and counterproductive idea, especially when the engine is already providing everything to play audio, and the Unreal audio team seems to be working hard to make the New Audio Engine better and better!

So, I began looking into parts of the engine, starting from the available tools (e.g. MediaPlayer), and I would like to ask if someone in the forum ever tried something like that, or knows what could be a good place to start.
From preliminary search, I was thinking about trying to implement something similar to the **UStreamMediaSource *object.
I would retrieve the audio stream buffer data through ffmpeg, decode it to have it available as uint8_t
buffer, and then finally feed it to a custom class which ideally should be nicely integrated with the engine.

EDIT:
I am actually looking into the AudioCapture Engine Plugin. I will probably try to create something similar, but instead of feeding the mic input to the audio buffer, I will pass the decoded audio. Hope it works out!

Any suggestions or comments? :wink:

Thanks!
Dario

Hi @turbogeeda
my name is Christian and I’m from italy, I’m currently developing an app for my university thesis using Unreal Engine, and one of the feature that I would like to implement is Video Streaming to another instance of the app over network (Both the apps are connected to the same router). Unfortunately I don’t have a lot of experience with Unreal and I don’t even know where to start :(. I’m sorry that I could not help with your issue but could you give me some advice on how I could stream video over the network? How do I implement fmpeg library in unreal? Is there a correct way to it?
Thank you! I’m waiting for your reply!
Christian.

Hi, you could take a look at RuntimeAudioImporter. Apart from its generic audio import functionality, it can also import audio data as a stream using the streaming sound wave. It supports various encoded audio formats, such as MP3, WAV, and OGG. All you need is a source of audio data in binary format (TArray<uint8> in C++ or an array of bytes in Blueprints).

1 Like