No announcement yet.

Playing realtime network audio in Unreal from custom stream source

  • Filter
  • Time
  • Show
Clear All
new posts

    Playing realtime network audio in Unreal from custom stream source

    This is my first post in this forum, so I would like to address a sincere "thank you!" to all the people around here developing and working with the engine, its plugins and writing about it

    I will go to the point of my post:
    in the next future, my team and I will need to play back RTP (or other) audio streams coming from a remote microphone for telepresence scenarios, within Unreal. We are capable of playing back an audio stream using ffmpeg through the ffplay tool from command line (tested on Linux and Windows).
    Also, we have been using a custom C++ integration of ffmpeg within Unreal in Windows for the last 1.5 years to play real time network video streams. It is not particulary polished, but it is giving nice results in terms of performance for our purpose, so I would like to take the same route for audio.

    Therefore, our plan would be that of using ffmpeg to decode RTP audio streams, and render them somehow within Unreal.
    That would be a big plus for us, since at some point we might need to work on the spatialization of the audio streams.
    Also, the alternative of introducing a custom audio renderer just to play those streams seems like a really counterintuitive and counterproductive idea, especially when the engine is already providing everything to play audio, and the Unreal audio team seems to be working hard to make the New Audio Engine better and better!

    So, I began looking into parts of the engine, starting from the available tools (e.g. MediaPlayer), and I would like to ask if someone in the forum ever tried something like that, or knows what could be a good place to start.
    From preliminary search, I was thinking about trying to implement something similar to the UStreamMediaSource object.
    I would retrieve the audio stream buffer data through ffmpeg, decode it to have it available as uint8_t* buffer, and then finally feed it to a custom class which ideally should be nicely integrated with the engine.

    I am actually looking into the AudioCapture Engine Plugin. I will probably try to create something similar, but instead of feeding the mic input to the audio buffer, I will pass the decoded audio. Hope it works out!

    Any suggestions or comments?

    Last edited by turbogeeda; 01-08-2020, 07:05 AM.