Announcement

Collapse
No announcement yet.

Media Framework Roadmap

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Media Framework Roadmap

    The new Media Framework in UE4 is becoming increasingly popular, and we're getting more and more questions and requests for it, so I figured I'd start a new thread to let everyone know what the plans are. Although this new feature is already working and usable, it is still somewhat of an exploratory effort, and some parts of it may change significantly throughout the next few months. Here are some of the main bullet points we have in mind (not in any particular order):
    • Performance optimization: The current implementation is not very efficient as it requires two frame buffer copies, one from the decoder thread into a temporary buffer, and then from the temporary buffer into the render thread. On some platforms, an additional copy from the render thread into the GPU memory may be required. We are currently refactoring some part of the engine's texture management to avoid these extra copies. That should significantly reduce the overhead, especially on mobile platforms.
    • Pixel format conversion on GPU: The player plug-ins are currently expected to return RGB frame buffers. This limits the number of video formats that can be supported, because some decoders cannot return RGB for certain videos. For example, we cannot currently play back H.264 encoded .mp4 files on Windows (only MPEG-4 is supported). One option is to convert the pixel format inside the plug-ins, but that is very slow. A better option is to do the conversion on the GPU. We are currently working on some changes that will allow us to do this and also support other video related tasks on the GPU, such as decoding the video itself.
    • Audio playback: Audio is currently just a small hack and not officially supported yet. We are planning high-quality audio playback support, including fast-forward, reverse, scrubbing, etc. There is no ETA for this feature yet, but we just hired a new audio programmer who may have time to work on this soon. In the meantime, a common workaround is to play audio using the existing Sound Editor, which works reasonable well for shorter video sequences that are not prone to getting out of sync.
    • HTML5, Linux & XboxOne support: We have not started working on player plug-ins for these platforms, but it is on the to-do list.
    • Media sources: Currently, media is specified using a URL, which can be either a file on the computer, or a location on the internet. We are planning to generalize this API into the concept of "media sources", which will also support other types of media, such as web cams, front and rear facing cameras in cell phones, dedicated video capture hardware and custom hardware and software integrations via plug-ins. This also includes per-platform settings, so you can have one source on Windows and another source on iOS, for example. Work on this will start very soon.
    • Sequencer integration: We are planning to integrate the Media Framework more tightly with Matinee and Sequencer in order to better support applications for the film and movie production industry. The goal is to allow content creators to integrate pre-recorded and real-time audio and video with real-time 3D graphics.
    • Cross-platform player plug-ins: It's hard to believe, but the ability to conveniently play the same media content on multiple platforms is still an unsolved problem in 2015 as far as free and/or open source solutions are concerned. The Media Framework currently implements per-platform player plug-ins that reuse the media playback functionality built into the operating system, i.e. WMF on Windows and AVF on MacOS. Each of these have different capabilities and limitations, which can result in a lot of extra pain for content creators. We are currently investigating VLC and VP8/VP9 integrations for a truly cross-platform solution.
    • YouTube support: We are currently investigating this well


    I hope that this answers some of the most common questions. If you have any other questions or suggestions, please don't hesitate to reply here, and I will update this post, thanks!

    Update 2016/09/08: There is now a Media Framework category on our public UE4 Issue Tracker. Feel free to vote on features you'd like to see first.
    Last edited by gmpreussner; 09-08-2016, 02:45 PM. Reason: added issue tracker link
    Gerke Max Preussner | UE4 Programming Consultant

    Logo
    Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
    Chat with me as gmpreussner on Discord.

    #2
    Cool! - I'm very interested in the 'dedicated video capture hardware' bit of media sources. I'm happy to compile from github and test any progress you make on this front. Let me know if I can help.

    Dan

    Comment


      #3
      Sounds good. When I get a bit of free time I'll wrap up my rtsp media streaming plugin and post it to github.

      Michael

      Comment


        #4
        Can't wait for this. It'll help excel a few of the projects iv'e been working on, mediaTexture has been the last hurdle! Great work as usual and let me know if you need any more of those 2kx2k video samples

        Comment


          #5
          Great Work Epic! We can not wait for these updates, Thank you!

          Comment


            #6
            Syphon could be a great addition to it.
            Many of us are using now UE4 for real time visual performances. Syphon is a must in such a case.
            Access to the different sound cards could be a good add-on too.

            Comment


              #7
              support other video related tasks on the GPU, such as decoding the video itself
              Awesome. Seems like the only reasonable way to playback 4K and HEVC video.
              Are you planning on doing this at a high level (DXVA2)? Or using more low level API's like nVidia's CUVID, Intel's QuickSync, etc?

              VP8/VP9 integrations
              Great! BSD licensed. Royalty Free. Google backed. What's not to like.

              VLC integrations
              Happy to hear this is still being considered. Yes it is LGPL licensed, but unless you can somehow absorb the costs of licensing/including a multi-codec decoder from CoreCodec or MainConcept/DivX, this seems like the best option. Perhaps turn it into a plugin that is not included with the UE4 default distribution but has to be downloaded separately by developers. That way there shouldn't be a need for a single line of LGPL code/headers/etc in the UE4 mainline.

              Comment


                #8
                Perhaps turn it into a plugin that is not included with the UE4 default distribution but has to be downloaded separately by developers.
                Yes, that's the plan. I will basically work on this in my free time as a side project. The repository will be here: https://github.com/ue4plugins/VlcMedia

                That being said, I haven't had much free time recently, so this will take a while. The main blocker with VLC right now is that the library is a giant pain in the neck to build for the various platforms. Pull requests are welcome
                Gerke Max Preussner | UE4 Programming Consultant

                Logo
                Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
                Chat with me as gmpreussner on Discord.

                Comment


                  #9
                  I submitted several commits that fix bugs with the handling of web URLs. See the related AnswerHub post for details.
                  Gerke Max Preussner | UE4 Programming Consultant

                  Logo
                  Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
                  Chat with me as gmpreussner on Discord.

                  Comment


                    #10
                    Great Work ! video capture card support is so useful in VR projects

                    Comment


                      #11
                      added preliminary vlcmedia support for linux

                      https://github.com/GeneralReality/Vl...commits/master

                      Comment


                        #12
                        Were the updates in the 4.8 preview? Cant see them listed!

                        Comment


                          #13
                          I seen this late but this is what I've been waiting for! This will be a good push for UE4 being more industry-friendly. I've encountered these limitations on a recent VR / Film project.

                          Comment


                            #14
                            I, too, would love to know if this made it into the 4.8 beta. My team is eagerly awaiting it, and if you're incorporating libvlc, you will be making our lives much easier!

                            Comment


                              #15
                              From my testing it seems its sligtly better, but one can assume they're still working on it

                              Comment

                              Working...
                              X