Announcement

Collapse
No announcement yet.

Media Framework Documentation for 4.5 Preview

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Media Framework Documentation for 4.5 Preview

    Hi all,

    We've been working on a new feature called the Media Framework for the past few weeks. It will add media playback functionality back into the Engine. A preview version is already in the Master branch, and it will also be available in 4.5. The feature is not quite production ready yet, as we're still working on a number of problems. Nevertheless, we'd like to provide some preliminary documentation for it, so you can start playing around with it and let us know how it works for you.


    Don’t we already have media playback from UE3?

    Unfortunately no, UE4 did not have this capability until now. In UE3 we used a third-party solution called Bink, but since we decided to make the entire source code of Unreal Engine 4 available to the public, we could, of course, no longer include Bink out of the box. Some of the old code pieces were still present in the Engine code base, but they were never implemented again for UE4. Rather than integrating one particular media playback solution in UE4, we decided to provide a generic API instead, which will allow anyone in the community to create their own player plug-ins for audio and video playback, including exotic formats and proprietary hardware. Of course, we will also implement some plug-ins at Epic.


    But wait, isn't there some movie playback code in the Engine already?

    We currently have the MoviePlayer system, but it is very limited and can only be used for playing startup movies while the Engine loads up. It cannot be used to play movies in-game, such as on the UI or on a static mesh in your level. The Media Framework will provide much more general media playback capabilities. Here is a list of the most important additions:
    • Engine & Slate agnostic
    • Localized audio & video tracks
    • Content Browser, Material Editor & Sound system support
    • Blueprint & UMG integration
    • Streaming media
    • Fast forward, reverse, pause & scrubbing
    • Pluggable players


    At some point we will use the Media Framework to also re-implement startup movie functionality. In the meantime, both the MoviePlayer system and the Media Framework will exist side by side.


    What exactly is the Media Framework?

    The Media Framework is largely a collection of C++ interfaces, a couple helper classes for common use cases and a media player factory that can be extended with so called Media Player plug-ins. All the interesting work of playing movies and other media files happens in the plug-ins. Currently, we have a player plug-in for Windows, which uses the Windows Media Foundation API under the hood. It is the most complete implementation, but it still has a few issues, particularly with certain movie formats, such as H.264 encoded .mp4 files. We are working on resolving these. We also have a plug-in for MacOS, which uses Apple's AV Foundation. Its feature set is currently quite limited, but it should also work on iOS (we haven't tested it yet). Work has begun on a player plug-in for Android.

    The framework itself is both Engine and Slate agnostic, which means that we can use it in pretty much any application, not just the game engine or the Editor. There are additional layers on top of the framework that provide media playback capabilities to other sub-systems, such as Engine, Blueprints, Slate and UMG. This should cover all expected use cases, such as in-game textures & UI, in-Editor video tutorials, and Marketplace videos.

    Click image for larger version

Name:	mediaframeworkarchitecture.jpg
Views:	1
Size:	9.6 KB
ID:	1138987

    Programmers can find the code for the Media Framework in the following location: /Engine/Source/Runtime/Media/
    The existing player plug-ins are located in the /Engine/Plugins/Media/ directory.


    How can I play media in the Engine?

    Most users, especially content creators will interact with media directly in the Content Browser. There are three new asset types that you can create there: MediaPlayer, MediaTexture, and MediaSoundWave.

    The MediaPlayer, as the name suggests, represents a player for a media source, such as a movie file on disk or a streaming media URL on the internet. Unlike UTextureMovie in Unreal Engine 3, MediaPlayers do not actually contain any data - they only store the path or URL to the media source. You can create a new MediaPlayer through the right-click menu in the Content Browser, or you can drag and drop a supported media file type into the Content Browser. What file types are supported depends on which player plug-ins you have installed:

    Click image for larger version

Name:	mediaplayerdragdrop.png
Views:	1
Size:	141.3 KB
ID:	1138989
    Important: As of this writing, local movie files on your coputer have to be located inside the /Content/Movies/ directory of your project. This is currently the only location that will be packaged correctly. We will improve this in the future.

    MediaPlayers can output various content streams. Currently supported stream types are audio, video and caption texts. If you double-click on a MediaPlayer object in the Content Browser, the Media Player Editor will open, and you can inspect the properties of the media source (note to programmers: the media asset editor is also the reference implementation for using the Media Framework with Slate):

    Click image for larger version

Name:	mediaplayereditor.jpg
Views:	1
Size:	32.0 KB
ID:	1138988

    Once you have a MediaPlayer, you can create from it a MediaTexture asset, which will allow you to extract a video stream. The MediaTexture allows you to pick a video stream (if more than one are available), and you can use it just like any other texture in the Engine. If the video doesn’t show on the texture but plays in the media asset editor then most likely the wrong video track is selected. You can change this by double-clicking the MediaTexture, which opens the texture editor.

    Click image for larger version

Name:	mediatexturecreation.png
Views:	1
Size:	90.6 KB
ID:	1138991 Click image for larger version

Name:	mediatexturecb.png
Views:	1
Size:	11.0 KB
ID:	1138992

    Click image for larger version

Name:	mediatexturelevel.png
Views:	1
Size:	523.6 KB
ID:	1138993

    MediaSoundWaves work exactly the same way for sound streams. They are still in a very early stage of development, however, and need a lot more work, so I’m not going to cover them just yet. Feel free to experiment with them on your own.


    Can I control media playback with Blueprints?

    Yes, all Blueprint based control of media playback is handled through the MediaPlayer class. It exposes a number of Blueprint functions and events that allow you to manipulate the media source, and to gather various properties and capabilities of the media itself:

    Click image for larger version

Name:	mediaplayerbp.png
Views:	1
Size:	30.5 KB
ID:	1138994


    Does it work with UMG?

    You bet it does. Since MediaPlayer provides a Blueprint API and UMG is able to consume Engine materials, adding media playback to UMG is straightforward. The ContentExample sample project now actually contains a demo level that implements an entire media player using UMG. The name of this level is UnrealMotionGraphics, which is currently located inside the project’s /Content/Maps/TestMaps/ directory:

    Click image for larger version

Name:	examplemap.png
Views:	1
Size:	422.0 KB
ID:	1138995

    Double-click the /Game/UMG/MediaPlayerWidget asset to open up the UMG Editor:

    Click image for larger version

Name:	mediaplayerwidget.png
Views:	1
Size:	190.7 KB
ID:	1138996

    There are various Blueprint scripts that implement the buttons, slider and text labels. You can test the UMG media player UI right in the Editor using ‘Play in Viewport’ and walking up to the first demo pod.

    Click image for larger version

Name:	mediaplayerumgbp.jpg
Views:	1
Size:	14.6 KB
ID:	1138998 Click image for larger version

Name:	mediaplayerumg.jpg
Views:	1
Size:	20.7 KB
ID:	1138997


    What about video capture devices, such as webcams?

    We are currently looking into the possibility of adding support for video capture hardware, such as webcams and movie production equipment.


    What else should I know?

    The Media Framework is not quite ready for production yet. Only Windows (and to some extent MacOS, and possibly iOS) is supported right now, and even there it can be a pain to find a working video format. Microsoft’s own formats, such as .wmv generally work without problems. Other common formats, such as .mp4 are more difficult. So far we only found the MPEG-4 codec to work reliably – H.264 is still having some problems, and .avi sometimes renders upside down. Microsoft is currently helping us to get these issues resolved and also to make everything work on XboxOne. We also still haven’t found a reliable way to embed caption text tracks inside media files, but hopefully that will soon be solved as well. Once everything is more polished we will release tutorials on how to properly create media content for your game.

    Android support is still a few days out, PS4 will follow afterwards. There are also some remaining usability issues with media playback that will be addressed over the next month or so. Audio tracks are not quite usable yet - audio will play, but may sound funny, and definitely won't work in reverse yet.

    In the coming weeks, the Media Framework API may change a bit in order to improve performance and usability, and to possibly accommodate other use cases, such as real-time video capture.
    Gerke Max Preussner | UE4 Programming Consultant

    Logo
    Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
    Chat with me as gmpreussner in #unrealengine on FreeNode IRC, or on UnrealSlackers.

    #2
    This is very good news! I appreciate all the work that has gone into allowing source code access without an NDA or anything like that, but still providing the kinds of features we need. Keep up the good work!

    Comment


      #3
      Hi there,

      So I see that packaging should work as long as I put the videos into the Content/Movie folder.

      I have the movie playing fine when I am doing play as Standalone game, however when I packaged the game all I see is a white screen, so I am not sure what I am doing wrong here.


      Here are everything I am doing
      -I got the 4.5alpha version from early last week.
      -Set Media Player URL to the exact full path on my local machine, inside the Content/Movies folder.
      -Set to Auto Play on Playback
      -Packaged development

      Thanks for the help! I am still looking into this.
      Last edited by TheWorkshopEnt; 10-02-2014, 10:50 PM.

      Comment


        #4
        Since you will eventually support video capture devices, will you also allow capturing a still frame from a video source such as a web cam or the camera on the mobile device? That would be a really useful features in creating avatars and user profile pics.

        Thanks for all the cool additions to the engine!

        Comment


          #5
          The timing on this feature could not be better. I've recently started working on a project that would rely heavily on playing video files on a surface. My question as someone who has yet to commit to an engine for the project is when will this be available? I looked for a release date on 4.5, and there does not seem to be one. Understandable of course, there must be a lot going into the release.

          Is it available to use in an alpha build? Or is there a rough idea of when 4.5 is slated to release?

          Thank you.

          Comment


            #6
            We are currently looking into the possibility of adding support for video capture hardware, such as webcams and movie production equipment.
            Interesting. Are you looking at adding some sort of live video chat via webcam during gameplay as a use case?

            If that is the case perhaps you should be looking into using WebRTC since it has a BSD Licensed, C++ API and is maintained by Google.
            http://www.webrtc.org/reference/native-apis

            Even if you are not looking at live video chat, you should still take a close look at the source code:
            https://code.google.com/p/webrtc/sou...ture%2Fwindows

            They have implemented video capture for multiple platforms using the respective native APIs. The code is all BSD licensed so I'm sure it will be of help even just as a reference.

            Comment


              #7
              This is what I've been waiting for since UE4 went 'live'. Thanks Epic!

              Comment


                #8
                Wow! This is great! Oh man! This can make creating tutorial videos much easier, and this could have some nice useful gameplay uses.

                One question. Is it possible to show a current scene going on the game?
                Musical Range VR | Musical Range Trailer | Pausing Game On Steam Overlay | Aperture Tag

                Comment


                  #9
                  Fantastic news!

                  Do we have support for alpha channels or does transparency work in a similar way to UE3 where you could run a alpha video in the opacity node to get transparency?

                  Comment


                    #10
                    Originally posted by TheWorkshopEnt View Post
                    Hi there,

                    I have the movie playing fine when I am doing play as Standalone game, however when I packaged the game all I see is a white screen, so I am not sure what I am doing wrong here.
                    I am having this problem as well. Any ideas?

                    Comment


                      #11
                      Would this work through streaming with a h.264 mp4 video hosted in Amazon S3?
                      ProBoon, Old-School FPS, Open Source & Tutorials
                      Support Project Boon on Patreon

                      On Steam Soon, Click Below

                      Comment


                        #12
                        Originally posted by thrillhouse900 View Post
                        I am having this problem as well. Any ideas?
                        Same issue here. Works fine with PIE but when I launch my media texture is white

                        Comment


                          #13
                          Audiocapture

                          Will audiocapture alone be supported so that we can plug mic input into an audioactor?
                          Last edited by Number47; 10-05-2014, 05:35 AM. Reason: Typos

                          Comment


                            #14
                            Where do you store your movie files?
                            Maybe this here is the issue (just guessing) ??

                            Important: As of this writing, local movie files on your coputer have to be located inside the /Content/Movies/ directory of your project. This is currently the only location that will be packaged correctly. We will improve this in the future.

                            Comment


                              #15
                              Originally posted by KVogler View Post
                              Where do you store your movie files?
                              Maybe this here is the issue (just guessing) ??
                              Yeah this is what I am doing. Tried a few different settings as well with no luck

                              Has anyone got it working in a built game? What codec works? Im using Mpeg-4 in a .mp4 with noooo luck

                              Comment

                              Working...
                              X