The new Media Framework in UE4 is becoming increasingly popular, and we’re getting more and more questions and requests for it, so I figured I’d start a new thread to let everyone know what the plans are. Although this new feature is already working and usable, it is still somewhat of an exploratory effort, and some parts of it may change significantly throughout the next few months. Here are some of the main bullet points we have in mind (not in any particular order):
Performance optimization: The current implementation is not very efficient as it requires two frame buffer copies, one from the decoder thread into a temporary buffer, and then from the temporary buffer into the render thread. On some platforms, an additional copy from the render thread into the GPU memory may be required. We are currently refactoring some part of the engine’s texture management to avoid these extra copies. That should significantly reduce the overhead, especially on mobile platforms.
Pixel format conversion on GPU: The player plug-ins are currently expected to return RGB frame buffers. This limits the number of video formats that can be supported, because some decoders cannot return RGB for certain videos. For example, we cannot currently play back H.264 encoded .mp4 files on Windows (only MPEG-4 is supported). One option is to convert the pixel format inside the plug-ins, but that is very slow. A better option is to do the conversion on the GPU. We are currently working on some changes that will allow us to do this and also support other video related tasks on the GPU, such as decoding the video itself.
Audio playback: Audio is currently just a small hack and not officially supported yet. We are planning high-quality audio playback support, including fast-forward, reverse, scrubbing, etc. There is no ETA for this feature yet, but we just hired a new audio programmer who may have time to work on this soon. In the meantime, a common workaround is to play audio using the existing Sound Editor, which works reasonable well for shorter video sequences that are not prone to getting out of sync.
HTML5, Linux & XboxOne support: We have not started working on player plug-ins for these platforms, but it is on the to-do list.
Media sources: Currently, media is specified using a URL, which can be either a file on the computer, or a location on the internet. We are planning to generalize this API into the concept of “media sources”, which will also support other types of media, such as web cams, front and rear facing cameras in cell phones, dedicated video capture hardware and custom hardware and software integrations via plug-ins. This also includes per-platform settings, so you can have one source on Windows and another source on iOS, for example. Work on this will start very soon.
Sequencer integration: We are planning to integrate the Media Framework more tightly with Matinee and Sequencer in order to better support applications for the film and movie production industry. The goal is to allow content creators to integrate pre-recorded and real-time audio and video with real-time 3D graphics.
Cross-platform player plug-ins: It’s hard to believe, but the ability to conveniently play the same media content on multiple platforms is still an unsolved problem in 2015 as far as free and/or open source solutions are concerned. The Media Framework currently implements per-platform player plug-ins that reuse the media playback functionality built into the operating system, i.e. WMF on Windows and AVF on MacOS. Each of these have different capabilities and limitations, which can result in a lot of extra pain for content creators. We are currently investigating VLC and VP8/VP9 integrations for a truly cross-platform solution.
YouTube support: We are currently investigating this well
I hope that this answers some of the most common questions. If you have any other questions or suggestions, please don’t hesitate to reply here, and I will update this post, thanks!
Update 2016/09/08: There is now a Media Framework category on our public UE4 Issue Tracker. Feel free to vote on features you’d like to see first.
Cool! - I’m very interested in the ‘dedicated video capture hardware’ bit of media sources. I’m happy to compile from github and test any progress you make on this front. Let me know if I can help.
Can’t wait for this. It’ll help excel a few of the projects iv’e been working on, mediaTexture has been the last hurdle! Great work as usual and let me know if you need any more of those 2kx2k video samples
Syphon could be a great addition to it.
Many of us are using now UE4 for real time visual performances. Syphon is a must in such a case.
Access to the different sound cards could be a good add-on too.
Awesome. Seems like the only reasonable way to playback 4K and HEVC video.
Are you planning on doing this at a high level (DXVA2)? Or using more low level API’s like nVidia’s CUVID, Intel’s QuickSync, etc?
Great! BSD licensed. Royalty Free. Google backed. What’s not to like.
Happy to hear this is still being considered. Yes it is LGPL licensed, but unless you can somehow absorb the costs of licensing/including a multi-codec decoder from CoreCodec or MainConcept/DivX, this seems like the best option. Perhaps turn it into a plugin that is not included with the UE4 default distribution but has to be downloaded separately by developers. That way there shouldn’t be a need for a single line of LGPL code/headers/etc in the UE4 mainline.
That being said, I haven’t had much free time recently, so this will take a while. The main blocker with VLC right now is that the library is a giant pain in the neck to build for the various platforms. Pull requests are welcome
I seen this late but this is what I’ve been waiting for! This will be a good push for UE4 being more industry-friendly. I’ve encountered these limitations on a recent VR / Film project.
I, too, would love to know if this made it into the 4.8 beta. My team is eagerly awaiting it, and if you’re incorporating libvlc, you will be making our lives much easier!
Hi
My team and I were working on implementing Vlc Movie Player plugin since we required it for our current project. We have gotten satisfactory results. We would like to share it with the community. I’ve already generated a pull request for the same.
, does this plugin work with MediaTextures? I am working on a small project with a friend and using WMF to stream to a MediaTexture. The downside is getting my source to a format WMF accepts. Having RTSP support would make things a lot easier.
Great work! Sorry for my late response - I have been travelling, was sick for a bit, and now totally slammed again. I will try to take a look at your submissions as soon as possible. I will most likely get back on Media stuff next week! I’m not sure if we can merge those changes into the Engine directly, license wise. That’s why I had created the separate ue4plugins Git repository. If you guys are interested, we can create a GitHub team and work on this together - I will send you an invite. Please PM me or send me an [EMAIL=“.preussner@epicgames.com”]email, thanks!
It should play H.264. I haven’t tested rtsp:// yet, but http:// works. This plug-in still has a lot of issues and is not finished yet (subtitles and audio are missing, etc). I do not recommend using it in production yet, but if you want to mess around with it, please do not hesitate to send pull requests
Please note that VlcMedia includes pre-compiled VideoLAN binaries that are licensed under LGPL. It is important that you consider the implications of this when distributing your game. Check the repository’s readme file details, and definitely also check out the LGPL license in the /ThirdParty/vlc directory.
what is the current status of the media framework integration with matinee? We would like to be able to scrub matinees along with the videos they contain, but it currently doesn’t seem to be supported. I suppose that it might require the blueprints that process media framework events to run while scrubbing? Is integrated matinee and media framework scrubbing functionality something that we can look forward to? Thanks!
I’ve encountered these limitations on a recent VR / Film project.