Dynamic/scrolling skybox from 360 video? Sync video/texture to movement?

Hi all,

First post! I’m new to UE4, but am learning and loving it, and I’m here to stay. I’ve been reading everything I can, but am struggling to come up with one essential piece for my current project.

I’m trying to come up with methodology to make a scrolling skybox/skysphere. It’s beyond my discipline, and I’m wondering if anyone has ideas on how to accomplish this or if something similar has been done.

In essence, I would like to use a 360 panorama video as a skybox, and I would like the video playback to be synced to the camera/character position. Therefore, as the character moves forward/backward, the video frames play forward/backward - similar to how a side-scroller works, but with a video/skybox.
I’ve played with the idea of using video textures, but don’t know how I would sync them to movement efficiently. I could also load/unload texture frames, but I don’t know how I could do that efficiently with so many frames (replace texture at every step).

Perhaps I’m going about this completely wrong? Any help/ideas/direction on how to proceed would be greatly appreciated! Thanks for reading.

This should be really simple to set up. Basically you need to rotate the texture coordinates of a cubemap using a blueprint that reads your player’s forwards movement. Remember cubemap has 3d coordinates.

Use Cameravector as the UVs (might have to multiply some axis by -1 to flip). then use a RotateAboutAxis node. Use your cameravector as position, use 0,0,1 for NormalizedRotationAxis, use a ScalarParameter for RotationAngle, use 0 for pivotpoint.

Then take the result of that and add it back to the original input camera vector (rotate about axis is intended for use in worldposition offset which has to be re-added for relative offset).

The rotation is not in degrees, 0 will be no rotation, 1 will be a full 360.

Now you just need to figure out an ideal rate of rotation that your game will add over time or will player motion.

I use this often for regular skies so that we can rotate the bright part of a cubemap to match a scene.

RyanB,

Thanks for the quick reply! Working with cubemaps is new to me, so please forgive my ignorance. I’m not sure I followed correctly, but are you describing actually rotating the whole texture, like a rotating ball? The problem for me is that I’m including more than just sky in the cubemap, I actually want to accomplish an effect more like side-scrolling, where the distant landscape that is filmed moves with the character position. If I could even figure out how to use a video for a side-scrolling background, and then adapt the methodology to additional axis, that would also be great. Again, forgive my floundering, I’m an art guy pursuing greater ambitions in UE4.

Yes, it would basically be like being inside of a spinning ball. That’s really all you can do in this case without making some kind of tiling orthographic image or series of images that tile together.

When I was suggesting this, I was concerned that if the rotation occurs too fast, it may be very obvious what is going on. It might look weird anyways. You could have one layer for the sky and then use flat tiling textures for a parallaxed panning city skyline or any combination of the two to get the effect you want.

If you want layers of parallax, you could make an alpha for your cubemap and rotate different layers at different speeds.

Indeed it looks like that is the route I may have to go. What I was hoping to do is, rather than creating a scroller with parallax layers, use a video that was shot to match the perspective. Because the video would be shot from the same perspective, it would automatically have perfect parallax. I was hoping for a photo-realistic effect without cutting layers from images. Kind of out-of-the-box, but I think it could accomplish a cool effect for my specific use.

Is there a way to tile together a series of images on a mesh without requiring a crazy amount of resources? Even on a single plane? So that every step equals several frames/images (or however many is feasible).

The problem you will have is that rotation causes 0 parallax. If you stand still and rotate around, you haven’t changed positions therefore all the elements far away will be in the same relative positions compared to near objects.

If you could shoot a video by actually panning the camera and make it loop by blending the beginning and end together, you should be able to get something cool.

*edit: I’m not sure where I got that you wanted to try rotation, maybe I misinterpreted your original post.

As far as the filming, it wouldn’t be panning, it would be “trucking”, which is when the camera moves horizontally (strafe left/right in video game terms). This way I would get the parallax effect of close and distant objects moving relatively more/less.

It seems to me I could do this if I could figure out how to effectively do one of the following (and then expand upon it later):

  1. Place a video on a plane that is locked to the character’s position, and sync the video playback with character movement.
  2. Create a sequence of images on a plane that is locked to the character’s position, and rapidly swap images based on character’s movement.

Any instructions on how to do this, or something similar, and I will be deeply in your debt! Seriously, I would be happy to trade hours of work for help implementing this.

Ah, trucking is a new phrase for me to learn :slight_smile:

I am not sure what the current state of video playback in the editor is unfortunately. This approach could work, but it may be faster to just create individual depth layers that tile and control the panning speed of each layer by the ‘distance’ you want to emulate. Also the video (or series of flipbooks) would probably take up much more memory than individual texture layers that tile and pan.

Your question 1 is easy enough though. you just have to attach a plane to the player that is behind the player and large enough to extend outside of the screen. Then for the texture UV’s you would use the screenposition node which would fit it perfectly to the screen. You should be able to pan the screenpositionnode as well, or just add values to the x-axis over time.

Thanks again for your replies, I’ll have to keep an eye out for information on embedding videos.

Using depth layers is certainly the most efficient and standard method, and it looks like I may have to go this route, but I’m hoping to go for a specific effect that is heavy on the cinematography/photography side. I’d be willing to sacrifice much of the available system resources if I could effectively incorporate this, as it is one of the primary elements I hope to incorporate, assuming the resources required aren’t beyond feasible. I guess I’m thinking of it kind of like a greenscreen in film, that plays each frame based on player coordinates, and is always an equal distance and axis from the character.
Maybe I can adapt the flipbook to work for my needs. Any idea if I could tie in the flipbook frames with player movement?

RyanB,

I am trying to do the same thing-- can you tell me what I’m doing wrong? I have a stitched together 360 panorama video that I’ve imported into UE4 and used to create a MediaTexture. I am now trying to replicate what you are describing but I keep getting this error: Imgur: The magic of the Internet. Any help greatly appreciated!