Computing dense optical flow with scene capture component 2D

[MadMartigan]

I am trying to compute dense optical flow in Unreal 5 using the Scene Capture Component 2D. What this means is that for each pixel in frame t-1, I want to know the vector (dx, dy) that will get me the matching pixel in frame t. I want this to account for both camera motion and the movement of meshes in the scene.

I have followed these older discussions to form my general approach (see link at bottom), but I’ve run into some snags where the output is not making sense. My general approach is to first create a custom material that uses SceneTexture:Velocity like so:

I then create a render target of format RGBA32F.
Lastly, I create a scene capture component 2D that targets my render target. It has the following settings:

Capture Source: Final Color (HDR) in Linear Working Color Space
Capture Every Frame: false
Capture on Movement: false
Always Persist Rendering State: true

In the PostProcessVolume for the Scene Capture, I add my material from above.

To render scenes, I have a C++ actor with the scene capture as a child, which I use to efficiently write out raw renders from the scene capture - I only write the R and G channels of the FLinearColor because I assume these are the motion vectors.

I am trying a basic experiment where I move forward in a straight line, which should give me flow vectors that point radially out from the center of the image. The raw image I get has very small min and max values:

MinX, MaxX: 0, 0.002677
MinY, MaxY: 0, 0.004555

If I visualize these vectors as a heatmap I get an image that has structure and information of some kind, but they don’t seem to point in the right direction:

I have also tried the DecodeVelocityFromTexture transformation that the velocity shader uses to map from (0,1) range to (-2,2), but this just maps the min/max values above to values close to -2 which doesn’t make sense:

const float InvDiv = 1.0f / (0.499f * 0.5f);

float3 V;
V.xy = EncodedV.xy * InvDiv - 32767.0f / 65535.0f * InvDiv;

I am a bit at a loss. Am I following the right approach for what I’m trying to do? Should I be writing a custom material shader instead? Note I am using Unreal 5.3.2.

Past discussion I have been following for reference:
Problem interpreting SceneTexture:Velocity Data - Development / World Creation - Epic Developer Community Forums (unrealengine.com)

[BananableOffense]

Couldn’t you just sample the velocity buffer directly and transform the world space velocity vectors into view space?

[MadMartigan]

Well, I’m a bit unclear on what the velocity buffer actually is - my understanding was SceneTexture:Velocity is intended to deliver motion vectors for computation of motion blur effects. In that case, aren’t the values supposed to already be in view space?

[BananableOffense]

Perhaps it’s already in view space by that point - I’ve never needed to use it directly - but yes you’re right it is mainly for motion blur and TAA. Regardless, is that not essentially a per pixel optical flow?

[jwatte]

@MadMartigan

1.0f / (0.499f * 0.5f);

This will be slightly larger than 4.0. You’ll probably want something like 1.0f/(0.501f*0.5f) or just hard-code something like 3.998f

[MadMartigan]

@jwatte

This will be slightly larger than 4.0. You’ll probably want something like 1.0f/(0.501f*0.5f) or just hard-code something like 3.998f

Why do you say that? I took that conversion snippet directly from the engine source code.

That aside, I’ve been digging deeper and feel I am close but not understanding something about the units of the velocity buffer. The values seem to be generally on the range (0,1). If I move to the right (scene is moving left) I get all zeros (which show up as white according to my optical flow color chart):

However if I move to the left (scene is moving right) then I get red values, which correspond to moving right according to my color chart, as expected.

So somehow negative vector values are getting clipped to zero. Okay, this makes sense if there is supposed to be a decoding step from (0,1) to (-2, 2). But most of my motion vector values are small (close to zero). This means they will just all map to around -2. How can I decode the values correctly so that I recover the original mix of positive and negative flow vectors? I am missing something here. Why do I get all zeros in the first image?

[jwatte]

@MadMartigan

Why do you say that? I took that conversion snippet directly from the engine source code.

That’s just math. Maybe the engine wants to saturate to -2…2 range. Maybe the engine has a bug, I don’t know. But your question was “why do my values go slightly outside -2 … 2” and the answer is “because 1.0 / (0.499 * 0.5) is greater than 4.0”

I haven’t looked at the motion buffer in detail – how are you reading them out? As 8-bit surfaces? Does it change if you read it out as 16-bit floats or 32-bit floats?

[BananableOffense]

Part of the issue is that a view space motion vector will operate as a 2D vector. It will only be RG, where R represents movement in horizontal screen space and G represents movement in vertical screen space. Your optical flow map is RGB. You would need to remap the RG vector into this RGB space.

Think of a 2 component vector where RG values represent the XY coordinates for plotting the vector. This will point in some direction in a circle, and have a given magnitude. That’s what a view space motion vector looks like.

Overlay that vector onto your key, and pick the color from that same coordinate, and use it.

One way you may be able to do this is to use the motion vectors as an offset for UV coordinates to sample the key image.
The other way would be to create a math formula that converts between these two ways of representing motion.

[MadMartigan]

@BananableOffense

Overlay that vector onto your key, and pick the color from that same coordinate, and use it.

Thanks, I understand this part. I’m confident with my color mapping code and don’t think that’s the issue. I am writing the motion vectors out as float32, just the R and G channels.

I figured out that one of my problems is I think I should be encoding the velocity in my material before writing it to the texture, this gets it into the proper range (0,1) without throwing away negative values:

When I load the dumped vectors from file, I decode them back to (-2, 2) and then multiply the X by image width and Y by image height to go from UV coordinates to pixels (which are the units I want for optical flow). Something like this (just a python experiment):

invDiv = 1 / (0.5 * 0.499)
mv_ = mv[i,j,:]
mv_ = (mv_ - (32767.0 / 65535.0)) * invDiv
mv_[0] *= width
mv_[1] *= height

Now, I have an experiment where I move backwards from one frame to the next:

In this case the corner of the blue cube moves one pixel to the right and almost zero vertically. That means I expect a motion vector something like (+X, 0) which maps to red in my color key.

Instead I got this: [-0.03310148 -0.34269208] which is basically (0, -Y), which results in a purplish color - so I don’t think there is a problem with my color mapping. The vector I’m getting from Unreal doesn’t seem right. I will keep digging, it’s entirely possible I made a dumb mistake somewhere.

My main assumptions:

• Velocity ranges from -2,2 and is measured in UV
• MV texture is an RG two-channel texture stored as float32, R is horizontal and G is vertical
• Motion vector measures the distance a pixel traveled from the previous frame to the current frame (dX, dY)

[BananableOffense]

So far everything seems correct to me, could be missing something too but I do think you’re on the right track. Interesting that the result was basically off 90 degrees counter clockwise. Is that true for other cases too? If you move -x is the output +y? If so maybe you just need to rotate the vector 90 degrees?

[MadMartigan]

I thought it might be something like that, but it’s a little more complex. To put it simply, if I sample the motion vectors on a horizontal line across the bottom of the image, this is what I expect to get:

But this is what I’m actually getting:

[BananableOffense]

Hm, that makes me think a UV is centered on (0,0) that needs to be centered on (0.5,0.5) or something along those lines. Just did a real quick mockup

I think this is right? Maybe not exact for your case but maybe it’ll help… Saturation node is just there to make the color more obvious. Texture sample is the flow field gradient.

[MadMartigan]

@BananableOffense

Hm, that makes me think a UV is centered on (0,0) that needs to be centered on (0.5,0.5) or something along those lines. Just did a real quick mockup

That’s interesting, how did you set that up? Applied the material to a sphere actor?

[BananableOffense]

No, it’s a post process material.

[MadMartigan]

Did you apply the postprocess material to a postprocess volume? I tried adding it to a sphere and just a got a transparent sphere that doesn’t change color when I move it. Sorry if this is a dumb question, I’m not extremely experienced with Unreal.

[BananableOffense]

Yes, the material is applied to a postprocess volume.

[MadMartigan]

Ok I was able to do something similar to your experiment by applying the post process material on a screen capture I have in the tool I’m making. It looks good as you say, but on closer inspection the colors do not really make sense. This shader definitely “looks legit” in the sense that it produces colors that get more intense the faster you move. But when you look at the actual direction of vectors it really does not make much sense.

Getting this (moving backward):

When I should be getting this:

It’s bizarre because it seems so close - the vectors decrease in magnitude toward the center which does make sense. Also if I move directly forwards instead of backwards the vectors are flipped 180 degrees as would be expected:

But the directions just aren’t right. For example the model’s legs are green - but he is shrinking towards the center (vectors are (-x, -y)) so the legs should be bluish. I’ve been playing with render target settings and scene capture settings but not getting anywhere. I’m thinking about maybe looking into a custom shader like the one provided by this plugin:

ydrive/EasySynth: Unreal Engine plugin for easy creation of synthetic image datasets (github.com)

Problem is that one does not capture dynamic mesh movement which I really would like. I’m not really a graphics expert and was hoping for an out of the box solution.

[BananableOffense]

I don’t have any color discontinuities like what you depict. Objects have consistent gradients for me. The setup seems to work on panning the camera or object. Although it seems like maybe something is off when pushing in/out. Not sure why that would be any different though.

[MadMartigan]

That’s my impression as well. Panning seems okay but when the camera moves in/out the vectors are weird. Thanks for the help. I’ll update here if I find out anything else.

[Study-is-happy]

Is there any luck on optical flow? I am also looking for a solution