Have you tried enabling Allow custom depth writes in the material?
It off by default since a couple of versions. Maybe it’s needed to write the distortion information? It’s a bit of a shot in the dark but some post process materials break without it on.
Sadly, I already used this method to create my effect.
But, this result in a black output for the render sequence and a Depth sequence without distortion in the output.
And what bothers me is that in the viewport on Buffer visualisation, there are no distortions, except in Lit view.
The MRQ neglects the effect while rendering your depth with the post process material (probably on purpose as any other post process effect would affect your depth anyways - didn’t look up the source). So what you would need to do would be to apply the same math in your DepthJo pp Material. For panini you could use this tutorial and the shader for your scene as well as for the depth UE4 Tutorial: Panini Projection - YouTube
Same goes for the tutorial from @3dRaven. The code should go into the DepthJo as well. Why your beauty is black in the sequence is probably something else to look at if you use the tutorial.
But we’r not going to use these methods. Indeed, we need to have a visually and physically correct perspective. The passes, only distort a rendering
We need an equivalent of a 180° lens.
Using raytracing rendering to get just optical flow and depth would have been perfect. We don’t want a cubemap stitch either.
Maybe, if you don’t have the solution, we will post another post in this direction.