Hi, I’m trying to use the Media Framework to play in app high resolution videos on a Galaxy Note 4. While the Qualcomm H.264 video decoder on this device has the ability to decode 4k video at 30 FPS, I’m getting far lower fps in my project.
Is the Media Framework plugin for Android currently using the hardware Qualcomm H.264 video decoder on this device or is it using some sort of software cross-device decoding?
Is there a way to enable the hardware accelerated decoding of videos for this specific device?
The movie support for Android uses android.media.MediaPlayer. There are two movie player systems; one that uses Slate (fullscreen) which the startup movies uses, and the other is the media framework plugin. The media framework plugin uses a shader to swizzle the rgba / bgra order before a glReadPixels into a memory buffer that is queued and copied back to a texture to be used by the material based on the playback rate. This adds a number of steps which for a 4k video would be expensive after the decoding.
Thank you for the explanation, Chris. Shall we expect any improvement to this in the near future?
Just for my personal understanding, if it wasn’t for the rgba / bgra conversion the video decoding would not impact the workload of the gpu, would it?
The biggest performance hit is the glReadPixels into a buffer which is queued for the media framework system to then turn around and upload to another texture. In future I’d like to see this step skipped and the FBO created be queued for direct use by the material instead, but the framework expects an RGB buffer. You can see the work done by MediaPlayer14.java in copyFrameTexture().
Thank you, I’ll give a look into it. So, right now I’m out of luck if I need to play hi res videos on a Note4? I’m trying to find a way to reproduce 360 videos in gear VR using UnrealEngine, do you think it is feasible at all, considering I should keep the global framerate (not the video one) at 60fps?
I’d say keeping 60 fps may be difficult if you need 4k resolution; I’d try scaling them down and see how it performs. Some changes are being made which will allow bypassing the extra copies, but it would be 4.10 at the earliest before this will likely be in for Android.
Currently I have to stay at around 480p if I want to keep 60fps in a very basic scene (only unlit materials, no translucent materials) with the GearVR. 720p causes already a big drop in the framerate (it goes as low as 15-20fps). Of course, this resolution is too low for a 360 video, likely I’d need at least 2048x2048 (it is totally feasible on this device with native applications, most of the 360 videos in oculus home are this resolution, even at 60fps - the video itself, not just the game)
Any update about this in 4.9? what is the maximum resolution for which I should expect a good performance in GearVR?
The hooks needed for optimization made it in 4.9, but reworking the Android media player code to use them will not. I will be looking at doing so for 4.10.
I’m also waiting for this feature, I’m happy to read some good news
Great, really looking forward for this!
Any update about this? Can we expect to see this in 4.10?
Also, in 4.9 I’m experiencing a few bugs related to the media player in Gear VR, like this one. Are those known issues?
Hi Chris & everyone !
So will we see this coming on 4.10? I truly need it !
Or is there any other solution that would have been found?
Thanks a lot
It is in 4.11. You can try it out in the 4.11 preview now.
Great, thank you, I will try it out soon! Do you know if also these issues got fixed with this release?
I haven’t had any issues playing mp4’s with or without GearVR in my testing for 4.11 with the performance improvements. I looked at your material setup; I usually have the texture connected to both base color and emissive and set shading model to unlit.
I am using 4.11 preview 4 and seeing similar performance degradation when playing back a 1024x1024 h.264 mp4 encoded video with bit rates variable between 5 and 7 mbs on my Note 4. Any suggestions on what I might due to improve the playback performance. When the video ends, fps goes from 15-20 to 60fps
I’ve been playing 1920x1080 h.264 mp4’s at 60fps on Note4 with Adreno. Which model do you have?
Hi Chris, what is the maximum resolution you were able to use on a note4 / gearvr and with which frame rate?
I looked into the low performance and discovered my optimizes were dropped out of the 4.11 branch by mistake during an integration. I’ve added them back in along with my fixes for SetRate. You can find these in this GitHub commit: https://github.com/EpicGames/UnrealEngine/commit/b151006b20094a34002a4f814b12d5a116c88fe8