Download

Draw texture from uchar *

Dear Community,

i’m working on a plugin that brings video streams into unreal and draws them on textures for display. I do have a working prototype but there are two things i wonder about.

I receive the video data frame by frame as a char* and currently i use this function to convert it to an TArray<FColor>:


for (int y = 0; y < VideoSize.Y; y++)
{
	for (int x = 0; x < VideoSize.X; x++)
	{
		int i = x + (y * VideoSize.X);
		Data*.B = frame->data[i * 3 + 0];
		Data*.G = frame->data[i * 3 + 1];
		Data*.R = frame->data[i * 3 + 2];
	}
}

then use GetData() to get it as an uint8* and pass it as the SrcData in the first method described here: https://wiki.unrealengine.com/Dynamic_Textures to update a texture.

Now this is working but the performance is not ideal so i wonder if this is best way to do this?

Another thing i would like to do is to draw to TextureRenderTarget2D instead of a Texture2D this would make it easier to use in the editor. But would this influence performance? And could someone point me in the right direction on how to do this?

I’m a c++ beginner and i really appreciate every help i can get.

Why is it not ideal? You mean low FPS? I used the code from this OpenCV integration tutorial (which is identical to yours I think) a while ago, with a much heavier marker recognition algorithm running behind, and everything was working fine.

Yes this is virtually the same i do but i have very large high fps video streams that indeed lead to an fps drop.

I don’t really know if there even is a faster way to do this, maybe that just what i get on a particular machine with this kind of video but i still wanted to ask the professionals if they notice something to improve.

And drawing to a TextureRenderTarget2D completely alludes me so any help here is also much appreciated.

If you think your solution is not fast enough, figuring out why is not as simple as looking at one piece of code.

This code chunk is only a small part of your process it sounds like, and may not be where your code is “slow”.

You should start by profiling your code as best as possible,
check out
https://docs.unrealengine.com/latest/INT/Engine/Performance/
You need to figure out if your gpu or cpu bound and continue your investigation from there.

You might need to add timers in your code if you want more specific performance metrics. see https://answers.unrealengine.com/questions/77017/how-to-get-high-resolution-time-stamp-for-timing-o.html

You can go all out trying to over optimize your small chunk of code like this (not tested, my not be syntactically correct)



int x, y, offsetY, offsetLocal, offsetFrame, sizeX = VideoSize.X, sizeY = VideoSize.Y; // Variables defined once.
WheverTheTypeOf"Data"is& localColor; // will reference one Data[offsetTotal] point
auto& frameColorData = frame->data; // Get a direct reference to data, once;
for (y = 0; y < sizeY ; y++)
{
        offsetY = (y * sizeX); // calculate once per row
	for (x = 0; x < sizeX ; x++)
	{
                offsetLocal = x + offsetY; 
		offsetFrame= offsetLocal*3; // Do the *3 once
                localColor = Data[offsetLocal ]; // Do the array lookup once
		localColor.B = frameColorData[offsetFrame + 0];
		localColor.G = frameColorData[offsetFrame + 1];
		localColor.R = frameColorData[offsetFrame + 2];
	}
}

The compiler might make all kinds of these substitutions for you, or it might not. The above changes might run slower, or run the same as yours. This is why you need to focus on profiling and figuring out where the time is actually being spent, and weather any changes you make actually run faster.

Edit: Here is another over optimized way, again this may not help performance at all, or it could totally make it worse



// This is a basic array copy from frame->data into Data
int i, offsetFrame, size = VideoSize.X * VideoSize.Y; // Variables defined once. 
WheverTheTypeOf"Data"is& localColor; // will reference one Data* point
auto& frameColorData = frame->data; // Get a direct reference to data, once;
for (i = 0; i < size ; ++i)
{
  offsetFrame = i*3; // Do the *3 once
  localColor = Data*; // Do the array lookup once
  localColor.B = frameColorData[offsetFrame + 0];
  localColor.G = frameColorData[offsetFrame + 1];
  localColor.R = frameColorData[offsetFrame + 2];

}

Thanks for your tips Sneppy and SilverB1rd,

i already started profiling that is why i thought the problem must be in this section, its the only thing that is new but now i think i’m doing something else wrong. I Just downloaded the webcam example “OpenCV integration tutorial” that Sneppy posted, followed the instructions carefully and only adjusted some small details to make it compile under 4.14 and i’m now testing with a normal low resolution webcam.

Surprisingly the results are the same it is extremely slow I have around 5 fps.

I’m going to do some more testing/profiling to see if i can find the problem.
I can also upload the project if someone could spare the time to show me how stupid i am…