OpenCV can't read VideoCapture

Development Programming & Scripting C++
, , , ,
1

I’m trying to read camera input using the built-in OpenCV library in UE5 but no matter what I do, I can’t seem to make cv::VideoCapture.read() return anything.
It works fine on the same machine with regular C++ with OpenCV 4.6.0 and the VideoCapture is definitely open and the camera turns on as expected.
Is there something about Unreal’s built-in implementation I need to know about? I can’t seem to find any info online about this.

This is what my header files look like:

#pragma once

#include "PreOpenCVHeaders.h"
#include <opencv2/core.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/imgproc.hpp>
#include "PostOpenCVHeaders.h"

#include "CameraReader.generated.h"

And my Plugin’s Build file:

PublicDependencyModuleNames.AddRange(
	new string[]
	{
		"Core",
		"OpenCVHelper",
		"OpenCV",
		// ... add other public dependencies that you statically link with here ...
	}
);

Using Windows 11 with Kinect 2.0.

2

Decided to replace VideoCapture.read() with VideoCapture.grab() and VideoCapture.retrieve(). I have discovered that at least VideoCapture.grab() is the problem as it returns false all the time.

3

After many days of finding a solution, I have finally found one.

  1. I made a copy of the OpenCV plugin from the Engine and added it to my project’s plugins.
  2. Downloaded the GStreamer Runtime and Devkit for Windows.
  3. Rebuilt OpenCV using CMake (making sure to include the provided Unreal module as an extra module), including most of the modules that Epic left out, namely GStreamer.
  4. Use GStreamer as my video API, by using:
VideoStream.open("autovideosrc ! videoconvert ! appsink", cv::CAP_GSTREAMER);
  1. Video and Camera now works.

Ultimately, I was unable to ever get anything other than GStreamer to work for cameras and I still don’t know why. They were all included in the plugin.
MSMF: :x: (opens camera but doesn’t grab frames)
DirectShow: :x: (infinitely holds up when trying to open camera)
OpenNI2: :x: (crashes when trying to open camera)

I did get FFMPEG to work for videos, but GStreamer ended up performing way better (it seems to take advantage of the GPU way better).
I also managed to get CUDA working perfectly by following normal OpenCV instructions.

The camera does read super slow and is only using the CPU. I will have to build GStreamer from source to install some plugins that can help with this.

4

What provided Unreal module are you talking about?

5

The OpenCV plugin that is built in to UE5 contains an additional OpenCV module called Unreal. It essentially binds OpenCV’s memory allocation to Unreal’s to prevent memory leakage.

6

When you rebuild OpenCV do you just build the opencv_world455.lib and opencv_world455d.lib libraries separately from Unreal or do you need to rebuild the whole engine?
In other words, can you unpack the process a little bit more?
I have built OpenCV many times for non Unreal projects, I am curious how the process differs when it’s integrated as an Unreal plugin.

Thanks for any help!

7

I built the OpenCV library as you typically would with CMake, with the only difference being the addition of the Unreal module I linked to it.

When an engine plugin of the same name is found in your project’s plugins folder, it completely overrides the engine’s version of the plugin.

So the solution is to copy and paste Unreal’s OpenCV plugin into your own project’s folder and replace the libs/dlls with the ones you just built with CMake.

(I had to build OpenCV using a different folder due to the max path size limit being reached fairly quickly in CMake when building in the Unreal Plugin folder).

8

Awesome! Thanks so much for the response!

9

OK, I am still having difficulty. I normally build opencv with cmake-gui. I am less familiar with using cmake in command line. How do you add the unreal module in this particular case?

10

I believe I also used the GUI but it was a while ago so I cannot say for certain exactly what I did. If it’s of any help, here’s the project I used it in. The OpenCV stuff is under Blink/Plugins/OpenCV.