The new Media Framework in UE4 is becoming increasingly popular, and we’re getting more and more questions and requests for it, so I figured I’d start a new thread to let everyone know what the plans are. Although this new feature is already working and usable, it is still somewhat of an exploratory effort, and some parts of it may change significantly throughout the next few months. Here are some of the main bullet points we have in mind (not in any particular order):
- Performance optimization: The current implementation is not very efficient as it requires two frame buffer copies, one from the decoder thread into a temporary buffer, and then from the temporary buffer into the render thread. On some platforms, an additional copy from the render thread into the GPU memory may be required. We are currently refactoring some part of the engine’s texture management to avoid these extra copies. That should significantly reduce the overhead, especially on mobile platforms.
- Pixel format conversion on GPU: The player plug-ins are currently expected to return RGB frame buffers. This limits the number of video formats that can be supported, because some decoders cannot return RGB for certain videos. For example, we cannot currently play back H.264 encoded .mp4 files on Windows (only MPEG-4 is supported). One option is to convert the pixel format inside the plug-ins, but that is very slow. A better option is to do the conversion on the GPU. We are currently working on some changes that will allow us to do this and also support other video related tasks on the GPU, such as decoding the video itself.
- Audio playback: Audio is currently just a small hack and not officially supported yet. We are planning high-quality audio playback support, including fast-forward, reverse, scrubbing, etc. There is no ETA for this feature yet, but we just hired a new audio programmer who may have time to work on this soon. In the meantime, a common workaround is to play audio using the existing Sound Editor, which works reasonable well for shorter video sequences that are not prone to getting out of sync.
- HTML5, Linux & XboxOne support: We have not started working on player plug-ins for these platforms, but it is on the to-do list.
- Media sources: Currently, media is specified using a URL, which can be either a file on the computer, or a location on the internet. We are planning to generalize this API into the concept of “media sources”, which will also support other types of media, such as web cams, front and rear facing cameras in cell phones, dedicated video capture hardware and custom hardware and software integrations via plug-ins. This also includes per-platform settings, so you can have one source on Windows and another source on iOS, for example. Work on this will start very soon.
- Sequencer integration: We are planning to integrate the Media Framework more tightly with Matinee and Sequencer in order to better support applications for the film and movie production industry. The goal is to allow content creators to integrate pre-recorded and real-time audio and video with real-time 3D graphics.
- Cross-platform player plug-ins: It’s hard to believe, but the ability to conveniently play the same media content on multiple platforms is still an unsolved problem in 2015 as far as free and/or open source solutions are concerned. The Media Framework currently implements per-platform player plug-ins that reuse the media playback functionality built into the operating system, i.e. WMF on Windows and AVF on MacOS. Each of these have different capabilities and limitations, which can result in a lot of extra pain for content creators. We are currently investigating VLC and VP8/VP9 integrations for a truly cross-platform solution.
- YouTube support: We are currently investigating this well
I hope that this answers some of the most common questions. If you have any other questions or suggestions, please don’t hesitate to reply here, and I will update this post, thanks!
Update 2016/09/08: There is now a Media Framework category on our public UE4 Issue Tracker. Feel free to vote on features you’d like to see first.