VR: Improving 4k 360 video playback in game

[CHADALAK1]

Hello Epic!

I was curious if there is any sort of future improvements for VR 360 video playback?

Just to be clear, I’m not talking about recording the video in UE4, I’m talking about playback in UE4.

360 video work has been growing very strong over the past year or two and quality is getting slightly better. The only engine I know that can play at super high quality is WorldViz’s Vizard engine.

I would like to see something like Unreal Engine be on top of the market for VR 360 video playback as well. From 4k - (future)8k resolution playback in engine.

[smbv1]

Hello, I also looking for 360 video playback inside UE4, to mix live action videos with engine 3D capabilities to make some interactive content applications. I hope Epic working this way to support 360 videos inside engine.

[xmak]

Hi.

Although there is currently no 360 video support in Unreal Engine, you can do it yourself. The way I did it goes like this…

1. Create a sphere mesh with normals pointing inside so that the video texture in Unreal is applied inside the sphere. You can do this by exporting a sphere mesh from Unreal and then doing invert normals in Blender (you can easily google how to do it), and then importing this new mesh in Unreal.

2. Open the new mesh asset in Unreal, and change it’s Import Translation so that the pivot point is in the middle of the sphere (it can be a little bit complicated to find the right translation, but it’s doable). The Import Translation can be found in the details panel when you open the mesh asset.

3. Create a new sphere actor based on this new mesh asset, scale it to a larger size, take an equirectangular 360 video, map it to the new sphere actor via Unreal MediaPlayer and MediaSource and play it (there’s a tutorial about playing video with Unreal MediaPlayer, google it).

4. When you want to display the 360 video, just set the location of the character to the location of the sphere actor (it will be in the middle of the sphere because in the step 1 we put the sphere pivot point to the center of the sphere), and restrict character movement so that it cannot go out of the sphere. The camera of the character used has to be at the center of the character (locaton 0,0,0) so thah when we put the character inside the sphere, the camera is at it’s center. As an alternative, you can have your own character blueprint in which you have the same inverted sphere mesh positioned on the location of the player camera.

NOTES:

• this works with equirectangular videos, so be sure to convert your 360 video to this format
• you can use the same principle to display 360 images
• with the 360 videos I used, I found that they are fliped horizontally when displayed, so I modified the media texture material (created when creating the MediaPlayer) to flip the texture horizontally (sshot below)

EDIT: You can skip the steps 1 and 2 (inverting the normals) if you make your media texture material two sided. Then you will be able to see the 360 video also on the outside of the sphere.

I hope this helps you and anyone else who’s trying to play 360 videos in Unreal.

Here’s the modified media texture material:

[Juss]

Works flawlessly now!

@xmak;689488

Hi.

Although there is currently no 360 video support in Unreal Engine, you can do it yourself. The way I did it goes like this…

1. Create a sphere mesh with normals pointing inside so that the video texture in Unreal is applied inside the sphere. You can do this by exporting a sphere mesh from Unreal and then doing invert normals in Blender (you can easily google how to do it), and then importing this new mesh in Unreal.

2. Open the new mesh asset in Unreal, and change it’s Import Translation so that the pivot point is in the middle of the sphere (it can be a little bit complicated to find the right translation, but it’s doable). The Import Translation can be found in the details panel when you open the mesh asset.

3. Create a new sphere actor based on this new mesh asset, scale it to a larger size, take an equirectangular 360 video, map it to the new sphere actor via Unreal MediaPlayer and MediaSource and play it (there’s a tutorial about playing video with Unreal MediaPlayer, google it).

4. When you want to display the 360 video, just set the location of the character to the location of the sphere actor (it will be in the middle of the sphere because in the step 1 we put the sphere pivot point to the center of the sphere), and restrict character movement so that it cannot go out of the sphere. The camera of the character used has to be at the center of the character (locaton 0,0,0) so thah when we put the character inside the sphere, the camera is at it’s center. As an alternative, you can have your own character blueprint in which you have the same inverted sphere mesh positioned on the location of the player camera.

NOTES:

• this works with equirectangular videos, so be sure to convert your 360 video to this format
• you can use the same principle to display 360 images
• with the 360 videos I used, I found that they are fliped horizontally when displayed, so I modified the media texture material (created when creating the MediaPlayer) to flip the texture horizontally (sshot below)

EDIT: You can skip the steps 1 and 2 (inverting the normals) if you make your media texture material two sided. Then you will be able to see the 360 video also on the outside of the sphere.

I hope this helps you and anyone else who’s trying to play 360 videos in Unreal.

Here’s the modified media texture material:

&stc=1

Thank you so much!

[CHADALAK1]

@xmak

Hi.

Although there is currently no 360 video support in Unreal Engine, you can do it yourself. The way I did it goes like this…

1. Create a sphere mesh with normals pointing inside so that the video texture in Unreal is applied inside the sphere. You can do this by exporting a sphere mesh from Unreal and then doing invert normals in Blender (you can easily google how to do it), and then importing this new mesh in Unreal.

2. Open the new mesh asset in Unreal, and change it’s Import Translation so that the pivot point is in the middle of the sphere (it can be a little bit complicated to find the right translation, but it’s doable). The Import Translation can be found in the details panel when you open the mesh asset.

3. Create a new sphere actor based on this new mesh asset, scale it to a larger size, take an equirectangular 360 video, map it to the new sphere actor via Unreal MediaPlayer and MediaSource and play it (there’s a tutorial about playing video with Unreal MediaPlayer, google it).

4. When you want to display the 360 video, just set the location of the character to the location of the sphere actor (it will be in the middle of the sphere because in the step 1 we put the sphere pivot point to the center of the sphere), and restrict character movement so that it cannot go out of the sphere. The camera of the character used has to be at the center of the character (locaton 0,0,0) so thah when we put the character inside the sphere, the camera is at it’s center. As an alternative, you can have your own character blueprint in which you have the same inverted sphere mesh positioned on the location of the player camera.

NOTES:

• this works with equirectangular videos, so be sure to convert your 360 video to this format
• you can use the same principle to display 360 images
• with the 360 videos I used, I found that they are fliped horizontally when displayed, so I modified the media texture material (created when creating the MediaPlayer) to flip the texture horizontally (sshot below)

EDIT: You can skip the steps 1 and 2 (inverting the normals) if you make your media texture material two sided. Then you will be able to see the 360 video also on the outside of the sphere.

I hope this helps you and anyone else who’s trying to play 360 videos in Unreal.

Here’s the modified media texture material:

Thanks for the info on creating a 360 video with UE4, but it’s not quite what I’m looking for.

Currently right now, 360 video is possible, but the video playback isn’t 60 fps (especially at 4K). I can create 360 videos no problem, but it’s the media framework’s quality when importing videos and playing them back.

[MothDoctor]

It’s quote from 4.18 Preview thread.
https://forums.unrealengine.com/unreal-engine/announcements-and-releases/1357546-unreal-engine-4-18-preview

Media Framework has been significantly updated and extended, and several media player plug-ins were largely rewritten in order to support upcoming advanced use cases, such as real-time video compositing, time synchronization, simplified workflows, capture hardware & video-over-IP support, online media streaming, integration with the Editor, and improved stability and performance. Parts of the API had to be redesigned in order to support these scenarios. Please note that some of this work, such as Sequencer integration, stereoscopic & 360 video support, and hardware accelerated 4K playback is still under active development.

So… it could be there in 4.19

[CHADALAK1]

@kjustynski

So… it could be there in 4.19

Awesome … But that wait thoooooooooo . I’m sure this will be worth the wait when 4.19 arrives :D. Thanks kjustynski

[anonymous_user_a6da3d73]

4.19 doesn’t solve this. The video playback for 4k 360 video in unreal still looks like garbage. Unreal is compressing the hell out of it for some reason. We used a VLC plugin for now, but this only looks marginally better. Whats the deal Epic?

[anonymous_user_3d131d741 (anonymous_user_3d131d74)]

hi guys!!! I used this for my 360 panomovie. And it worked… and render time was about 3 minutes for each frame, but when I try do it again - rendertime = 18 minutes. What do you think?

[anonymous_user_af75f483]

Was dealing with a similar issue re: video quality. In case you haven’t already, make sure in your render settings you are using forward shading and have MSAA enabled. Makes a huge difference in quality

[spacegojira]

Why would you even want 4k resolution in VR? The screen is way too small to even display that many pixels.

[sdfsidf]

HI.

I’m a bit new to unreal engine and im trying to get this to work, however im a bit lost.

I have the inverted normals via blender working, but I’m doing something wrong because i’m still getting the video to play outside

[Markom3D]

@spacegojira

Why would you even want 4k resolution in VR? The screen is way too small to even display that many pixels.

It is not 4k just infront of you, it is the whole video side that wraps around you. So what you area looking at isnt even 1k

[anonymous_user_3a734a63]

Has anyone found a solution to the video quality? I have the 360 material working but there is a lot of distortion

[anonymous_user_3a734a63]

@-beyond

Was dealing with a similar issue re: video quality. In case you haven’t already, make sure in your render settings you are using forward shading and have MSAA enabled. Makes a huge difference in quality

Please explain this, is this for capture or playback

[DARA059]

Were there any improvements in 4.22 in relation to this issue?

[cgiteach]

None seen by us in 4.23

[papalagi]

Hello everybody,

i’ve used the solution in this thread to play 360 videos with Unreal 4.23.1 on the Oculus Go

the video has been made with GoPro Fusion in 5.2K resolution

the very bad part is that there’s a big gap between the quality i can obtain playing the video with Oculus own player and inside Unreal, looks like Unreal arbitrarily lowers the resolution of the video

any idea on that?

have a nice day

[papalagi]

Elaborating on my previous post…

I’ve scaled the video down to 4096x2048, hoping power of 2 values could help, but i still experience jagged edges and suboptimal resolution

[papalagi]

Elaborating some more:

in my case the problem was in the UV Flipping code proposed in this thread, i dunno why but it messed with the video display somehow

straightforward solution: import the sphere in Blender, flip the UV horizontally, reimport in Unreal…

now it plays fine 4k video, i’m triyng 5k next