Stereo Panoramic Player

[UNAmedia]

Hi! Epic has been super-efficient, and the update is already available!

To upgrade: open the Epic Games Launcher > Unreal Engine > Library, under the UE4.24 launch icon you should see an exclamation mark near the "Installed Plugins" label (if not, restart the Launcher). Click the label, then the Update button near the Stereo Panoramic Plugin icon.

Version 1.1.0 changes (UE4.24-only):

• Added support for a new Opaque rendering mode: is more efficient to render (especially on low-end mobile devices) but objects in the surrounding 3D scene that overlap the panoramic sphere will appear inside. This mode supports rendering custom 3D meshes inside the panoramic sphere out-of-the-box. See 'Blend mode' for more details.
• Refactored the plugin materials, using now UE4 Instanced Materials, to simplify end-user customizations.
• UPanoramicBlueprintFunctionLibrary::TracePanoramicWidget() can now be used to trace widget components created with UPanoramicBlueprintFunctionLibrary::NewPanoramicWidget(). This simplifies detecting interactions with custom UMG widgets in a APanoramicSphere actor.

The new Opaque rendering mode is very helpful if you want to show 3D meshes inside the panoramic sphere (here more details), or if you don't have any surrounding 3D scene and want maximum performances (e.g. on Mobile platforms). If you need to support generic surrounding 3D scenes, then you should continue using the standard rendering mode.

We added detailed documentation on How to render 3D objects inside the Panoramic Sphere when using the standard rendering mode.

We added detailed documentation on How to manage custom interactive UMG widgets.

We hope this update will help with your work.

[UNAmedia]

Hi everyone! We submitted the new Version 1.1.0 to Epic for review. Following your requests, we added a new rendering mode (it's more efficient to render and supports out-of-the-box the rendering of 3D meshes inside the sphere) and a simplified management of custom UMG hotspots. The new version is backward compatible, so you should be able to update without any major issue. More details in the next days.

[UNAmedia]

Hi again!

As im finalizing development of my application, I noticed that the Stereo Panoramic Player doesnt 'unload' the videos from previous stages and keeps them in memory. Im only using 4 videos, but since its running on an Oculus go, it cant keep 150MB of videos loaded into the ram at all times. The third video only plays the audio, because it cant load it in anymore. Is there a way to unload videos after going to the next scene?

Thanks!

Kind regards

Hi! If you're using the APanoramicDirector class, then you can use the OnStageChanged delegate to be notified when the active stage changes. From there you can unload any previously played video.

In alternative (e.g. if you are using the low-level APanoramicSphere class) you can use the delegates from the used UE4 UMediaPlayer class (e.g. OnEndReached or OnMediaOpened).

Bests.

[RobbeVlaeminck]

Hi again!

As im finalizing development of my application, I noticed that the Stereo Panoramic Player doesnt 'unload' the videos from previous stages and keeps them in memory. Im only using 4 videos, but since its running on an Oculus go, it cant keep 150MB of videos loaded into the ram at all times. The third video only plays the audio, because it cant load it in anymore. Is there a way to unload videos after going to the next scene?

Thanks!

Kind regards

[UNAmedia]

Hi,

Thanks again for the quick response. Changing the stereomaterials to opaque instead of translucent solved my issue. Thanks a lot!

Thanks for the update!
All this complex setup with translucent materials is needed because if the 3D scene contains translucent objects, they pop-in in the panoramic sphere (due to how the deferred rendering pipeline renders translucent objects). Without translucent objects, everything is simpler.

[RobbeVlaeminck]

Hi,

Thanks again for the quick response. Changing the stereomaterials to opaque instead of translucent solved my issue. Thanks a lot!

[UNAmedia]

Hi again.

Im currently developing an application with your plugin, and I have some additional questions

The performance cost of the translucent materials (even with opacity set to 1) is pretty high, and since custom depth does not work on mobile, you get these nasty artifacts since you cant cull the triangles.

- is there a way to put the sort priority of the sphere back to 0? as long as its rendered in an empty level i dont think it should cause any issues.

-another solution would be to use a custom stencil to mask out the overlapping meshes and somehow 'force render' them over the translucent image sphere, but I dont really have a clue on how to do that.

In the pictures attached you can see ES3.1 (mobile renderer) vs SM5 (windows) Any help would be appreciated

Hi,

the suggested solution works also on mobile renderers and has been verified as supported by the plugin. From your screenshot it seems you hand doesn't have the correct described setup. If you need to work with other solutions (e.g. using custom depth buffers or stencil masks), you'll likely need to modify the plugin source code. Here few hints to let you start working with the sources:

- to change the sphere sort priority, you can edit the APanoramicSphere constructor, where the TranslucentSortPriority is set. In alternative, you can call GetComponentByClass() on the sphere actor searching for the type UProceduralSphereComponent or UProceduralMeshComponent, on the resulting component you can then invoke the SetTranslucentSortPriority() method.

- you can find the original translucent materials applied to the panoramic sphere enabling, in the Content Browser, the View Options "Show Engine Content" and "Show Plugin Content", then navigating to the "StereoPanoramicPlayer Content" folder. You can try overriding the original materials there or, in alternative, you can edit the method APanoramicSphere::GetSuitableMaterial() to return a your new custom material.

[RobbeVlaeminck]

Hi again.

Im currently developing an application with your plugin, and I have some additional questions

The performance cost of the translucent materials (even with opacity set to 1) is pretty high, and since custom depth does not work on mobile, you get these nasty artifacts since you cant cull the triangles.

- is there a way to put the sort priority of the sphere back to 0? as long as its rendered in an empty level i dont think it should cause any issues.

-another solution would be to use a custom stencil to mask out the overlapping meshes and somehow 'force render' them over the translucent image sphere, but I dont really have a clue on how to do that.

In the pictures attached you can see ES3.1 (mobile renderer) vs SM5 (windows) Any help would be appreciated

[UNAmedia]

Hi,
Sorry for the late response, but I got it to work! The issue was indeed the media texture framework. For some reason it does not support 4096*4096 videos :/ quite a shame, but thanks for the help! Everything is working great now

Thanks for the update! I'm glad you solved Often the most critical part for panoramic rendering are the used assets. In particular on mobile, devices could have many limitations/constraints.

[RobbeVlaeminck]

Hi,
Sorry for the late response, but I got it to work! The issue was indeed the media texture framework. For some reason it does not support 4096*4096 videos :/ quite a shame, but thanks for the help! Everything is working great now

[UNAmedia]

Alright, I bought the plugin, and its very easy to use, but Im having some slight issues. I added a stereopanoramic video to test things out, and while it works great in editor, the video is not playing on the Oculus Go. All I can see is a green texture(I added a screenshot), while the sound is working just fine. Both the ES2 and ES3.1 preview show the video just fine in the editor as well.
The video is also in the correct 'Movies' folder, so it packages correctly.
(The pink in the screenshot is just a rectangle to test the translucency sorting)

Hi! The plugin doesn't interact with the video, it simply uses the specified Media Texture (updated by the UE4 Media Player) when drawing the panoramic sphere. So I suspect the problem could be with the decoding of the video on the Oculus Go (or the format of the Media Texture when running in the Oculus Go). Here some useful links. You should be able to verify the setup simply applying your Media Texture to a plane in your scene (without using the Stereo Panoramic Player plugin), then you should see your video playing on the 3D plane.

[RobbeVlaeminck]

Alright, I bought the plugin, and its very easy to use, but Im having some slight issues. I added a stereopanoramic video to test things out, and while it works great in editor, the video is not playing on the Oculus Go. All I can see is a green texture(I added a screenshot), while the sound is working just fine. Both the ES2 and ES3.1 preview show the video just fine in the editor as well.
The video is also in the correct 'Movies' folder, so it packages correctly.
(The pink in the screenshot is just a rectangle to test the translucency sorting)

[UNAmedia]

Hi! Thanks for the fast response!

So since its displaying the motion controllers with the translucent sort priority, Im assuming its rendering the stereoscopic images with a translucent material as well. Mobile GPUs dont like translucent materials all that much, so I was wondering what the actual performance impact of this is. How well does it run on an oculus quest? can it hit the targeted 72 hz while displaying videos?

Thanks again!

Kind regards,

Robbe

You are right, the panoramic sphere uses a translucent material. But the rendering of the panoramic sphere is usually very cheap. We tested our Demo Application on both iOS and Android low-end devices (e.g. iPhone 6S) without noticing performance issues (with both images and videos). Usually - due their dimensions - the used assets are the more critical part for the final performances. Unfortunately I don't have numbers of the plugin running on the Oculus Quest. To keep an high frame rate, ensure to use texture that are POT and are using the best format for the hardware; for videos, ensure to encode them so that the decoding can be done in hardware.

(Just a clarification: the plugin doesn't render the motion controllers, if you want to render them then you have to follow the above instructions.)

[RobbeVlaeminck]

Hi! Thanks for the fast response!

So since its displaying the motion controllers with the translucent sort priority, Im assuming its rendering the stereoscopic images with a translucent material as well. Mobile GPUs dont like translucent materials all that much, so I was wondering what the actual performance impact of this is. How well does it run on an oculus quest? can it hit the targeted 72 hz while displaying videos?

Thanks again!

Kind regards,

Robbe

[UNAmedia]

Hi! Thanks for asking. Answers below.

- Does this support point and click interaction with a motioncontroller (if so, which ones) or only with gazing?

By default the Panoramic Director (the high-level API) doesn't use the motion controllers while in "panoramic" mode. But adding support to interact with the motion controllers is simple:

• use the method UPanoramicInteractionComponent::PushCustomWorldTargetDirection() on the APanoramicDirector::InteractionComponent property to report where the motion controller is pointing to (e.g. in your Pawn's Tick() method);
• enable the Interaction Input Action Mode (here for details) and, in the Input Action Name, specify the motion controller input you want to use to interact with the panoramic elements.

If instead you are using the low level API, all interactions are managed manually. See https://www.unamedia.com/ue4-stereo-...ps_interaction .

Due to how UE4 renders transparent objects and how motion controller meshes are managed, visualizing the motion controllers inside the panoramic sphere requires two more steps:

• you must assign Translucent materials to your motion controllers (if you're using a custom mesh it's trivial, otherwise you can rely on the Display Mesh Material Overrides property of the Motion Controller Component);
  
• you must set the property Translucent Sort Priority, of the static mesh component rendering the motion controllers (by default it should be the DisplayComponent property - but could change depending on the VR system used), to a value greater than 100. E.g. with the VIVE:

- Does this support the Oculus Go and Quest?

The plugin doesn't have any dependency on the used VR system, so it works on any device supported by UE4.