Media Framework poor quality rendering on Gear VR

[toddsdk]

When rendering a 4K monoscopic video with media framework on Gear VR, the video quality drops dramatically as soon as any other object starts being rendered on top of the video. Even a debug message being printed on screen will trigger the quality drop, and the quality will stay low even after the message has disappeared. Any other kind of overlay will also do it, for example, leaving the Default Virtual Joysticks enabled (default for Mobile projects).

Here are the steps to reproduce the bug:

1. Using UE 4.16, create a new project with the following parameters: C++, Basic Code, Mobile/Tablet, Maximum Quality, No Starter Content;
2. Create a new Empty Level;
3. Create a new MediaPlayer asset (check to create both additional assets: SoundWave and MediaTexture);
4. Create a Material from the MediaTexture;
5. Edit this newly created Material: Check the “Two sided” option; Connect the Texture Sample output to the “Emissive Color” input of the final material node; Save and close it;
6. (Optional) Import a high-poly Sphere static mesh with equirectangular UVs;
7. Add a Sphere (or the imported one) to the Level, reset its Location to default (0, 0, 0);
8. Apply the Material (created from MediaTexture) to the Sphere;
9. (Optional) Add the SoundWave Asset to the Level, reset its Location to default (0, 0, 0);
10. Add a Player Start to the Level and reset its Location to default (0, 0, 0);
11. Place a high resolution (4K) 360º monoscopic video file inside the /Content/Movies folder;
12. Add a new FileMediaSource asset and set its “File Path” to the video file from the previous step;
13. Open the Level Blueprint and set the Event Graph like the following: On BeginPlay, Open Source using the MediaPlayer and the FileMediaSource; On InputTouch Pressed, Print String (any string), making sure “Print to Screen” is checked;
14. Save all the assets and the level;
15. Edit the Project Settings:
16. In Maps & Modes, set the created Level as the “Game Default Map”;
17. In Packaging, add the created Level to the “List of maps to include in a packaged build”;
18. In Engine - Input, clear the “Default Touch Interface” (should display “None”);
19. In Platforms - Android, press the “Configure Now” button, set both Minimum and Target SDK Versions to 21 and enable “Configure the AndroidManifest for deployment to GearVR”;
20. In Engine - Rendering, disable “Mobile HDR” (This should prompt an engine restart to apply the settings. Do so right away);
21. Package the project as Android ASTC;
22. Install the package in a Samsung GearVR-compatible phone (used Galaxy S7 in my tests) and run it;
23. You should be able to watch the video in the intended quality.
24. Touch the Gear VR touchpad (or the phone’s screen if running with Developer mode on), and a message will be printed on the screen while the video will instantly have its quality lowered, becoming much more pixelated and/or with a lot of aliasing.

If you repeat the steps above using UE 4.14 or 4.13 instead, the video quality will remain the same (instead, what may happen in this case is an upside-down video being rendered until the first touch, which is a known bug, already fixed in recent versions).

The following repos contain two projects created using the above steps. The first one uses UE 4.14, in which the bug does not occur, and the other uses UE 4.16, in which it does. Important to note that this bug was not present in 4.13, but in 4.15 it already was.

[Bitbucket][1]

[Bitbucket][2]

This bug happens regardless of video codec (tested with H.264 and HEVC) or resolution (tested 4096x2048 3840x2048, among others).

Researching online, this issue seems to be very similar (if not the same root cause) to the following recounts:

[GearVR - Video Texture - Aliasing? - Mobile - Unreal Engine Forums][3]

[[GearVR] Media Texture Aliasing/Filtering Problem - XR Development - Unreal Engine Forums][4]

Finally, here are two images to better illustrate the problem:

UE4.14 without Bug (expected video quality):

UE4.16 with Bug (poor video quality):

[PattyVanFolter]

I have the same Problem, not only with gear, but with all my android devices.

[anonymous_user_55d7803c]

I’m getting the same problem too. I’ve been slowly brute force testing it, and it looks like it also happens when there’s a translucent/alpha composite material in the level.
So basically anything that interfaces with UI/Transparency triggers it by the look of it.

[toddsdk]

Were you able to debug down to the engine code? Maybe if we narrow it down to more specific parts of the code, the Epic staff will be able to take a better look into this bug! I’ve tested with 4.17 and it’s still happening, unfortunately.

[anonymous_user_55d7803c]

I’m on a BP project, so that was about as far as I got. I’ve since given up and downgraded to 4.14 in order to get it to work.
Interesting (weird) note on that though - I found that 4.14 also goes pixelated if you turn the visibility of the on screen virtual touch joypad controllers down to zero. Might have a shared issue with the bug on later versions of the engine.

[jwtrp]

Joining the club! Having the same problem since a while.

[toddsdk]

I’ve tested these steps in 4.18 Preview 3, and the bug seems to be gone. There is no aliasing, the video is not pixelated or anything anymore. Let’s hope it stays this way in the official release of 4.18.

However, a new bug has appeared, as the video colors are very bright and wrong, as the image below shows. By pressing the touchpad (which prints a string), the colors return to normal, and the video quality stays as expected.

[Chris_Babcock]

Which device model are you seeing this with? Can you include a device logcat?

[toddsdk]

Hello Chris, thank you for the interest!
The device I’m working with is a Samsung Galaxy S7 SM-G930F (Mali-T880 MP12)
I have attached the logcat below, as you requested.
By the time of this line:
10-10 09:24:12.246 10866 10881 D UE4 : [2017.10.10-12.24.12:246][144]LogBlueprintUserMessages: [NewMap_C_0] Hello
the video is back to normal colors.

[anonymous_user_a5054dd4]

Hey . I am trying to work around the fact that in 4.17, UE4 crashes 50% of the time when media framework is engaged on a 360 video somewhere in the level + VR preview mode is exited (HTC Vive). Is 4.18 more stable? Also - could you walk me through how you made a 360 video in 4.18? It seems that things have changed and my old method does not work anymore.

[toddsdk]

Hey Chris, apparently this color bug is still present in the release version of 4.18.0. Any chance of getting a fix in the next update?

[toddsdk]

The instability is still present in 4.18 when running on my PC, but on Android, the videos are stable. I have never had any crashes on Android.
For the 4.18, there has been some changes in the workflow of working with the Media Framework. Some of them are: you need to change the texture sampler type in your Media Texture Material to External; delete your old Media Sound Asset from level and from the project altogether and instead add a media sound component to an actor in the level; link your media player inside the media texture and inside the new media sound component. There are some official Epic videos on youtube that walk you through all these changes in the framework, if you need more guidance

[La_Frontera]

Hi, i have been working around the same problem for days

On the Android device, there was a significant quality drop playing the 360 video compared with the iOS version,

Finally I got a quality boost by modifying the material asset, activating Mobile->Use Full Precision

Hope it helps for you too

Link where I found that: Native Android Mediaplayer on Galaxy Note 4 - Mobile Development - Unreal Engine Forums

[anonymous_user_1dafbe0f]

Hey ,

I’m still having this issue as of 4.18.3 and 4.19.0 Preview 4. I had thought the way to fix it was to have an unbaked reflection capture in the scene but after seeing this post I realised it is ANY print string that fixes the colours! What do you think it is about a print string that causes the colours to correct themselves? How is a string of text rendered that it seems to reset something?

I’m having to hide a bit of black text as my video starts as a work around but it would be great to understand what is going on a little better.

[eatbuckshot]

I’ve had this same issue testing across 10 Galaxy S7’s: 6 x SM-G930T, 2 x SM-G930A, 1 x SM-G930V, and 1 x SM-G930F. the T (T-mobile),A (AT&T),V (verizon) variants that use the Snapdragon 820 SoC with Adreno 530 GPU does not exhibit this inverted? color issue. While the one SM-G930F does (Mali-T880MP12 on the Exynos 8890 SoC). The issue did go away once I started adding 2D Paper Sprites to the Level I believe. However I still have the issue exhibit itself when using a custom material that is emmisive only and sets the color in my case to a light blue/cyan for a 2DPaperSprite using the default white square texture… It will appear yellow on the Exynos

[knotff]

It looks like a render pass is missing or does not complete fully complete when playing a MediaTexture, so printing something to screen will refresh it and thus concluding the render pass.

The issue seems to lie if you use External on the Texture Sampler, on the Material Options . If you change it from External to Color on the Material everything works fine, but unfortunately this option does not work for MediaTextures I’m afraid.

The solution (well, it’s a workaround) is to “force” a pass by adding a primitive to the scene, so that if forces the camera to render this object and thus updating the MediaTexture on the objet.

On the image bellow I’ve added a Sphere to the right (off screen) of the player on start. Notice that the screen starts with the bug, but when I pan the camera to the right, the MediaTexture gets fixed when viewport renders the sphere.

http://www.giphy.com/gifs/NUtv8jU1wq5s4HjP7M

So the workaround to play the video without seen this added primitive is to place a Sphere around the camera/player. The player will not see the Mesh, because of the Normal direction. Furthermore this will increase the draw call very so slightly.

Other things I tested that failed:
• Update any Material properties (emission, textures, etc)
• Neither Forward nor Deferred renders worked.
• Adding Post-Process filters/materials
• Cacheing the video files
• Anything involving light or texture on World/Project/Material Settings.

This is valid for:
UE4.19.2
Samsung S7 (gearvr)