Is there possibility to display projection instead of camera view. How?

Development Rendering
1

Hello!

I am involved in project where we are making like AR, but instead of moving display as camera, there will be a moving viewer and a static big screen where fake space, similar to existing behind display
will be showed. Since the screen is static, there is need to get like projection of what we see on fake 3d screen and make anamorphic/stretched back to corners of screen transfer to view of this projection.

For example like here on following pictures

243533980bfa7c75934537a251bfe7bc789dc87f.jpeg
This is possible scene with virtual display before the space or room behind it

625a679c95fea3c3124a13878607ea6218e01352.jpeg
And here’s the anamorphic stretch of projection from point of view witch must be displayed on output of “game”.

Could You, UE4 Gurus, tell is it possible to make such renderings/morphs of such reverse projections? Maybe there are some
plugins or planed scripts for that. Thank You!

2

I’ve done this before in Unity–the way to do it was to change the camera projection matrix
I copied this from somewhere, can’t remember since it was like a year ago and while it’s C# it might help you get on the right track:


    using UnityEngine;
    using System.Collections;
     
    public class CameraProjection : MonoBehaviour
    {
        //eye
        public static Transform eye;
	
		public GameObject Tracker;
     
        //screen
        //public GameObject screenObj;
        public GameObject screenBottomLeftObj;
        public GameObject screenBottomRightObj;
        public GameObject screenTopLeftObj;
     
        void Update()
        {
			transform.position = Tracker.transform.position;
            //eye tracking
            eye = transform;
            Vector3 eyePosition = eye.position;

     
            //screen tracking
            Vector3 screenBottomLeft = screenBottomLeftObj.transform.position;//lower left of screen
            Vector3 screenBottomRight = screenBottomRightObj.transform.position;//lower right of screen
            Vector3 screenTopLeft = screenTopLeftObj.transform.position;//upper left of screen
     
            //off-axis camera projection
            Matrix4x4 m = Projection(eyePosition, screenBottomLeft, screenBottomRight, screenTopLeft, camera.nearClipPlane, camera.farClipPlane);
            camera.projectionMatrix = m;
        }
     
        static Matrix4x4 Projection(
            Vector3 eye,
            Vector3 screenBottomLeft,
            Vector3 screenBottomRight,
            Vector3 screenTopLeft,
            float nearClip,
            float farClip)
        {
            Matrix4x4 projectionMatrix = Matrix4x4.zero;
     
            float left, right, bottom, top;
            float screenDistance = screenBottomLeft.z - eye.z;
     
            left = (screenBottomLeft.x - eye.x) * nearClip / screenDistance;
            right = (screenBottomRight.x - eye.x) * nearClip / screenDistance;
            bottom = (screenBottomLeft.y - eye.y) * nearClip / screenDistance;
            top = (screenTopLeft.y - eye.y) * nearClip / screenDistance;
     
            projectionMatrix = PerspectiveOffCenter(left, right, bottom, top, nearClip, farClip);
     
            return projectionMatrix;
        }
     
        static Matrix4x4 ProjectionCAVE(
            Vector3 eye,
            Vector3 screenBottomLeft,
            Vector3 screenBottomRight,
            Vector3 screenTopLeft,
            float nearClip,
            float farClip)
        {
            Matrix4x4 projectionMatrix = Matrix4x4.zero;
     
            Vector3 screenXVector;
            Vector3 screenYVector;
            Vector3 screenZVector;
     
            screenXVector = screenBottomRight - screenBottomLeft;
            screenYVector = screenTopLeft - screenBottomLeft;
     
            screenXVector.Normalize();
            screenYVector.Normalize();
     
            screenZVector = Vector3.Cross(screenXVector, screenYVector);
            screenZVector.Normalize();
     
            Vector3 eyeToBottomLeft;
            Vector3 eyeToBottomRight;
            Vector3 eyeToTopLeft;
     
            eyeToBottomLeft = screenBottomLeft - eye;
            eyeToBottomRight = screenBottomRight - eye;
            eyeToTopLeft = screenTopLeft - eye;
     
            float eyeDistance;
            eyeDistance = -Vector3.Dot(eyeToBottomLeft, screenZVector);
     
     
            float left, right, bottom, top;
     
            left = Vector3.Dot(screenXVector, eyeToBottomLeft) * nearClip / eyeDistance;
            right = Vector3.Dot(screenXVector, eyeToBottomRight) * nearClip / eyeDistance;
            bottom = Vector3.Dot(screenYVector, eyeToBottomLeft) * nearClip / eyeDistance;
            top = Vector3.Dot(screenYVector, eyeToTopLeft) * nearClip / eyeDistance;
     
     
            // glFrustum implementation
            projectionMatrix = PerspectiveOffCenter(left, right, bottom, top, nearClip, farClip);
     
            Matrix4x4 rotationMatrix;
            rotationMatrix = Matrix4x4.zero;
     
            rotationMatrix[0, 0] = screenXVector[0];
            rotationMatrix[0, 1] = screenYVector[0];
            rotationMatrix[0, 2] = screenZVector[0];
            rotationMatrix[0, 3] = 0;
     
            rotationMatrix[1, 0] = screenXVector[1];
            rotationMatrix[1, 1] = screenYVector[1];
            rotationMatrix[1, 2] = screenZVector[1];
            rotationMatrix[1, 3] = 0;
     
            rotationMatrix[2, 0] = screenXVector[2];
            rotationMatrix[2, 1] = screenYVector[2];
            rotationMatrix[2, 2] = screenZVector[2];
            rotationMatrix[2, 3] = 0;
     
            rotationMatrix[3, 0] = 0;
            rotationMatrix[3, 1] = 0;
            rotationMatrix[3, 2] = 0;
            rotationMatrix[3, 3] = 1;
     
            projectionMatrix = rotationMatrix * projectionMatrix;
            //projectionMatrix = projectionMatrix * rotationMatrix;
     
            Matrix4x4 translationMatrix;
            translationMatrix = Matrix4x4.zero;
            translationMatrix.SetTRS(-eye, Quaternion.identity, Vector3.zero);
     
            projectionMatrix = translationMatrix * projectionMatrix;
     
            return projectionMatrix;
        }
     
        static Matrix4x4 PerspectiveOffCenter(float left, float right, float bottom, float top, float near, float far)
        {
            float x = 2.0F * near / (right - left);
            float y = 2.0F * near / (top - bottom);
            float a = (right + left) / (right - left);
            float b = (top + bottom) / (top - bottom);
            float c = -(far + near) / (far - near);
            float d = -(2.0F * far * near) / (far - near);
            float e = -1.0F;
            Matrix4x4 m;
            m = Matrix4x4.zero;
            m[0, 0] = x;
            m[0, 1] = 0;
            m[0, 2] = a;
            m[0, 3] = 0;
            m[1, 0] = 0;
            m[1, 1] = y;
            m[1, 2] = b;
            m[1, 3] = 0;
            m[2, 0] = 0;
            m[2, 1] = 0;
            m[2, 2] = c;
            m[2, 3] = d;
            m[3, 0] = 0;
            m[3, 1] = 0;
            m[3, 2] = e;
            m[3, 3] = 0;
            return m;
        }
    }
3

If I understand correctly what you’re looking to achieve is portal rendering. Except instead of rendering to a surface you want to render it to the framebuffer.

There’s an old rendering thread on setting up a portal using blueprints. It might be possible to render to a texture then as a post process effect overwrite the framebuffer with that texture.

4

Oh! Thanks Guys for answers! But would You please help more specificaly as I can’t find exact solution/tutorial to make this work properly and without spoiling something.
I suppose that some C++ programming required as I could not find nay plugin for that. I understood that there’s some problem with the blurring or lagging image.

5

Thank You for answer. But because of lack of detailed instructions, that tread is not helping. I would be very glad if someone would make some tutorial. I think It would be useful not only for me.

6

There’s not going to be a tutorial for something like that. You’d need to have some programming skill to do that.

7

For example, there’s a scripts/plugin available solving projection matrix,

but there’s a lack of explanations/instructions where exactly put them and activate.
It’s normal that if You share a script/programm it meant to be used by those who are not programmers
or not so advanced. I’ve learned that there must be some modifications made in engine or classes,
but without experience in engine(not programming) structure, I am afraid to spoil all thing.

So in mentioned “patch”(?) there’s a
MyGameViewportClient.cpp
MyGameViewportClient.h
Utils.h

No other files or comments about what I should do with them. Should I just put them in folder or
modify some of existing files. And what result should I expect. Whether there will be additional properties
in some of objects/classes so I can control and adjust to specific project…

I would also agree to buy commercial solution - plugin for that, but there’s no such at the moment.
Would You maybe write one? I bet You could earn something as interactive VR or AR things became popular.
Problem is that I must solve that soon. There’s no UE4 experts in my country and those who familiar with this topic
seem very busy.