Hello,
First off, I think the PixelStreaming plugin is a GODSEND. It’s extremely useful for many different applications and makes streaming basically anything you can store in a render target very easy and fast. (as long as you’re cool with the conversion of that data to something H264 or some other video codec can stream).
Here’s what I’m doing:
For a college design team, I’ve been developing a custom robot simulator for prototype mars rovers using Unreal Engine 5. I have basically everything for the sim finished. Sensors, drive train, telemetry, and cameras are all functional on a chaos vehicle that is rigged to an export of our rovers CAD model. The rover is being driven autonomously from an external codebase using the local machines network stack. The only thing left to make this sim fully functional is the ability to stream the virtual cameras back to the external codebase via the PixelStreaming plugin.
Here’s the challenge:
The virtual camera has three different things it needs to send to the external codebase.
- A normal RGB video feed
- A depth image (0-255 scaled)
- A depth measure (distance at each pixel in cm)
I have already setup a camera blueprint with a SceneCapture2DComponent for the depth and RGB camera capture. Each of these three things is stored in their own RenderTarget, and I’m using the PixelStreaming plugin to stream the render targets to my codebase. The RGB image, and depth image work perfectly.
However, the depth measure is a little bit trickier as the depth values can not be easily packed into the RGB values of the render target and sent through a H264 stream without compression artifacts and other data loss issues. I found this research paper which solves my exact problem and it outlines an algorithm that can be used to pack uint16 depth values into a YUV image such that they are resilient to YUV420P’s chroma sub-sampling and H264’s compression. I have written python code that implements the paper and it works surprisingly well even for low bitrates. That python code can be found here if you’re interested. I’ve tested it and this code works as expected.
I need your help!
I have written HLSL code that implements the exact algorithm shown in the python code that is known to be working. But when I connect to the WebRTC connection that the PixelStreaming plugin creates, and decode the received image, the depth values are not accurate.
I believe this is because the algorithm described in the paper is designed to have the L, Ha, and Hb values packed directly into the Y, U, and V channels of the YUV420P image that the H.264 codec streams. (H.264 and many other codec require the use YUV formats). But when writing to the render target, I can not do this. I can only use LinearColor, which expects RGB values from 0-1.
I think what’s happening is that the L, Ha, and Hb values stored in RGB colorspace in the rendertarget are INTERNALLY being converted to YUV420P colorspace when passed to the PixelStreaming plugin. This makes sense, but this messes up the encoded bits and therefore garbles the depth map when decoded on the other side.
I need some advice on how I can directly access the YUV values or convert store the L, Ha, and Hb values in the rendertarget so that the values will be maintained when passed to the PixelStreaming plugin.
I have tried backwards converting the L, Ha, and Hb from YUV to RGB, then storing that in the render target, but it didn’t help.
Here’s my HLSL code:
// Normalize the depth value.
float normalizedDepth = (depth + 0.5) / w;
// Period for triangle waves.
float p = np / w;
// L(d): Linear mapping for low-resolution depth.
float L = normalizedDepth;
// H_a(d): Triangle wave function 1.
float Ha = fmod(L / (p / 2.0), 2.0);
Ha = Ha <= 1.0 ? Ha : 2.0 - Ha;
// H_b(d): Triangle wave function 2 (phase-shifted by π/4).
float Hb = fmod((L - (p / 4.0)) / (p / 2.0), 2.0);
Hb = Hb <= 1.0 ? Hb : 2.0 - Hb;
// Convert YUV to RGB
float Y = L * 255.0;
float U = Ha * 255.0;
float V = Hb * 255.0;
float R = Y + 1.402 * (V - 128);
float G = Y - 0.344136 * (U - 128) - 0.714136 * (V - 128);
float B = Y + 1.772 * (U - 128);
RED = R / 255.0;
GREEN = G / 255.0;
BLUE = B / 255.0;