OPENCV with UE5 in 2024 - Beginner mistakes

Development Programming & Scripting
1

Hello,
I’m trying to get the 4.10 version of openCV working with UE5.3.
Most of the information I’m finding is related to an older version of the engine and doesn’t seem to work for me.

This is the tutorial series im following
Jordan Duncan - UE4 & openCV Part 1

I have some getting started questions. Please help if you can (hopefully someone from Epic Games too.)

  • If I use a newer version of ocv (4.10) do I still need to enable the plugin that ships with ue5 (4.5)
  • If I just cant get ocv to work via Third Party plugin process. Is there any code or example file that enables and uses the web camera using the inbuilt ocv plugin

I will describe what im doing to get the recent downloaded version of opencv linked in the ThirdParty folder l. Hopefully someone can spot some mistakes.

  1. Download a version of opencv that i am interested in 4.10 opencv download

  2. Create a UE5 VC++ project in MS Visual Studio 2022

  3. Create a ThirdPary folder and paste the necessary files there :
    ThirdParty\OpenCV\includes\opencv2
    ThirdParty\OpenCV\Libraries\Win64\opencv_world4100.dll
    ThirdParty\OpenCV\Libraries\Win64\opencv_world4100.lib
    ThirdParty\OpenCV\Libraries\Win64\opencv_videoio_ffmpeg4100_64.dll

  4. Populate the Build.cs file with the following code.
    I have a feeling im not doing something right in this step wrt to linking the path in the local project folder where the ThirdParty folder and opencv files are ? My local project is in the D drive.

// Copyright Epic Games, Inc. All Rights Reserved.

using System.IO;
using UnrealBuildTool;

public class opencv_alt : ModuleRules
{


    private string ThirdPartyPath
    {
        get { return Path.GetFullPath(Path.Combine(ModuleDirectory, "../../ThirdParty/")); }
    }

    public bool LoadOpenCV(ReadOnlyTargetRules Target)
    {
        // Start OpenCV linking here!
        bool isLibrarySupported = false;

        // Create OpenCV Path 
        string OpenCVPath = Path.Combine(ThirdPartyPath, "OpenCV");

        // Get Library Path 
        string LibPath = "";
        bool isdebug = Target.Configuration == UnrealTargetConfiguration.Debug && Target.bDebugBuildsActuallyUseDebugCRT;
        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            LibPath = Path.Combine(OpenCVPath, "Libraries", "Win64");
            isLibrarySupported = true;
        }
        else
        {
            string Err = string.Format("{0} dedicated server is made to depend on {1}. We want to avoid this, please correct module dependencies.", Target.Platform.ToString(), this.ToString()); System.Console.WriteLine(Err);
        }

        if (isLibrarySupported)
        {
            //Add Include path 
            PublicIncludePaths.AddRange(new string[] { Path.Combine(OpenCVPath, "includes") }); // original text reads as "Includes" i.e with capital I . my folder name reads "includes" i. without capitalized i

      
// EXPLICIT LIBRARY NAME PublicAdditionalLibraries.Add("D:\\behram\\UE5_dev\\opencv_5_alt\\opencv_alt\\ThirdParty\\OpenCV\\Libraries\\Win64\\opencv_world4100.lib");

//Add Dynamic Libraries            PublicDelayLoadDLLs.Add("D:\\behram\\UE5_dev\\opencv_5_alt\\opencv_alt\\ThirdParty\\OpenCV\\Libraries\\Win64\\opencv_world4100.dll");// EXPLICIT DLL PATH
            PublicDelayLoadDLLs.Add("D:\\behram\\UE5_dev\\opencv_5_alt\\opencv_alt\\ThirdParty\\OpenCV\\Libraries\\Win64\\opencv_ffmpeg320_64.dll");// EXPLICIT DLL NAME
        }

        PublicDefinitions.Add(string.Format("WITH_OPENCV_BINDING={0}", isLibrarySupported ? 1 : 0));
        //        Definitions.Add(string.Format("WITH_OPENCV_BINDING={0}", isLibrarySupported ? 1 : 0));

        return isLibrarySupported;
    }


    public opencv_alt(ReadOnlyTargetRules Target) : base(Target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
               PublicDependencyModuleNames.AddRange(new string[] 
{ 
"Core", 
"CoreUObject", 
"Engine", 
"InputCore", 
"EnhancedInput", 
"Renderer", 
"RenderCore", 
"RHI", 
"RHICore",
 "D3D12RHI", 
"RenderCore",
 "Media", 
"MediaAssets" }
);
        LoadOpenCV(Target);
    }
}
  1. When I create a simple c++ Actor and include opencv headers. I get the warning, “cannot open source file ThirdParty/OpenCV…” and so on.
// Fill out your copyright notice in the Description page of Project Settings.

//OPENCV INBUILT PLUGIN MODULE DISABLED

#pragma once

#include "CoreMinimal.h"
#include "GameFramework/Actor.h"
#include "ThirdParty/OpenCV/includes/opencv2/core.hpp"
#include <ThirdParty/OpenCV/include/opencv2/highgui/highgui.hpp>
#include <ThirdParty/OpenCV/include/opencv2/imgproc.hpp>
//#include "PreOpenCVHeaders.h"
//#include "OpenCVHelper.h"
//#include "PostOpenCVHeaders.h"
#include "opencv5_alt_test.generated.h"

UCLASS()
class OPENCV_ALT_API Aopencv5_alt_test : public AActor
{
	GENERATED_BODY()
	
public:	
	// Sets default values for this actor's properties
	Aopencv5_alt_test();
	void TestOpenCV();

protected:
	// Called when the game starts or when spawned
	virtual void BeginPlay() override;

public:	
	// Called every frame
	virtual void Tick(float DeltaTime) override;

};

What am i not understanding about the process so far ?

Thank you so much for your help
b

2

Update 01
Learned a ton over the course of a month. Thanks to @gabaly92 & @hardingnick001 foe helping me along the way.

I’m using the inbuilt OpenCV plugin for now. I will move onto compiling the latest version of OpenCV after I am comfortable with using the inbuilt one.

Here’s the code to get the webcam to work. It will open the webcam and save a frame to disk.

Header file

// Fill out your copyright notice in the Description page of Project Settings.

#pragma once
//#with_opencv = 1;

#include "CoreMinimal.h"
#include "GameFramework/Actor.h"
#include "PreOpenCVHeaders.h"
#include "OpenCVHelper.h"
#include "opencv2/unreal.hpp"
#include "opencv2/opencv.hpp"
#include "opencv2/core.hpp"
#include "opencv2/ml.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"
#include "PostOpenCVHeaders.h"

#include "test_blur.generated.h"

UCLASS()
class TEST_BLUR_TWO_API Atest_blur : public AActor
{
	GENERATED_BODY()
	
public:	
	// Sets default values for this actor's properties
	Atest_blur();

	// OPENCV FEILDS
	cv::Mat frame;
	cv::VideoCapture stream;
	cv::Size size;

	void TestOpenCV();
	void OpenCVGausianBlur();
	void TestOpenCVWebcam();


protected:
	// Called when the game starts or when spawned
	virtual void BeginPlay() override;

public:	
	// Called every frame
	virtual void Tick(float DeltaTime) override;

};

Source file

// Fill out your copyright notice in the Description page of Project Settings.


#include "test_blur.h"

// Sets default values
Atest_blur::Atest_blur()
{
 	// Set this actor to call Tick() every frame.  You can turn this off to improve performance if you don't need it.
	PrimaryActorTick.bCanEverTick = true;

}

void Atest_blur::TestOpenCV()
{
	// TEST OCV VERSION & READ & WRITE CAPABILITY
	GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::White, TEXT(CV_VERSION));
	
	
	cv::Mat img = cv::imread("D:/path/to/your/folder/test_write.png", cv::IMREAD_COLOR); 

	if (img.empty())
	{
		GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::White, TEXT("img is empty"));
	}
	else
	{
		int image_width = img.rows;
		int image_height = img.cols;
		GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::White, FString::Printf(TEXT("dimensions width = %d,height = %d"), image_width, image_height));


}

void Atest_blur::OpenCVGausianBlur()
{

    // READ an image 
    cv::Mat img_color = imread("D:/path/to/your/folder/test_write.png", cv::IMREAD_COLOR);

    auto outout_path = "D:/path/to/your/folder/";
    

    // BLUR CODE
    // Apply identity filter using kernel
    cv::Mat kernel1 = (cv::Mat_<double>(3, 3) << 0, 0, 0, 0, 1, 0, 0, 0, 0);
    cv::Mat identity;
    filter2D(img_color, identity, -1, kernel1, cv::Point(-1, -1), 0, 4);
  

    // Blurred using kernel
    // Initialize matrix with all ones 
    int ksizex = 12;
    int ksizey = 12;

    cv::Mat kernel2 = cv::Mat::ones(ksizex, ksizey, CV_64F);

    // Normalize the elements
    kernel2 = kernel2 / 25;
    cv::Mat img;
    //cv::Mat blur_img;
    filter2D(img_color, img, -1, kernel2, cv::Point(-1, -1), 0, 4);
    
    
    
    GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::White, FString::Printf(TEXT("blur size x = %d,y = %d"), ksizex, ksizey));

    imwrite("D:/path/to/your/folder/test_blur_kernel.png", img); // Use png or JP2 if needed since jpeg is not supported
    //https://github.com/EpicGames/UnrealEngine/blob/d4ab788a7191352128aa43720cff5b7dc25abcac/Engine/Plugins/Runtime/OpenCV/Source/ThirdParty/OpenCV/include/opencv2/cvconfig.h#L72
    



}

void Atest_blur::TestOpenCVWebcam()
{
    GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::White, FString::Printf(TEXT("TESTING WEBCAM  ")));

    //cv::Mat webcam_image;
    cv::VideoCapture vid_capture(0, 700);//CAP_DSHOW

	int frame_width;
	int frame_height;

	// Initialise frame matrix
	cv::Mat webcam_frame;
	cv::Mat grey;
	

	//BLUR SETUP
	int kernel_size = 10;
	cv::Mat kernel2 = cv::Mat::ones(kernel_size, kernel_size, CV_64F);
	// Normalize the elements
	kernel2 = kernel2 / (kernel_size * kernel_size); //5*5
	cv::Mat img_blur;


    // Print error message if the stream is invalid. Check if camera is opened
    if (!vid_capture.isOpened())
    {
        GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, FString::Printf(TEXT("Error opening video stream or file  ")));
    }

	while (vid_capture.isOpened())
	{

		
		// Initialize a Boolean to check if frames are there or not
		bool isSuccess = vid_capture.read(webcam_frame);

		// If frames are present, show it. 
		if (isSuccess == true)
		{

			//framenum++;

			//GET VIDEO WIDTH & HEIGHT
			frame_width = vid_capture.get(3);//CAP_PROP_FRAME_WIDTH
			frame_height = vid_capture.get(4);//CAP_PROP_FRAME_HEIGHT

			//WRITE a frame to disk


			//GREYSCALE
			cvtColor(webcam_frame, grey, 6);

			//BLUR
			filter2D(grey, img_blur, -1, kernel2, cv::Point(-1, -1), 0, 4);

		
			//WRITE BW
			cv::imwrite("D:/path/to/your/folder/test_bw.png", grey);
			//WRITE BLUR
			cv::imwrite("D:/path/to/your/folder/blur_kernel.png", img_blur);
			
		}

		// If frames are not there, close it
		if (isSuccess == false)
		{
			GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, FString::Printf(TEXT("Video camera is disconnected  ")));
	
			break;
		}

		
	}


}

// Called when the game starts or when spawned
void Atest_blur::BeginPlay()
{
	Super::BeginPlay();
	GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::White, TEXT("constructor test"));
	
	//TestOpenCV();
    //OpenCVGausianBlur();
    TestOpenCVWebcam();
	
}

// Called every frame
void Atest_blur::Tick(float DeltaTime)
{
	Super::Tick(DeltaTime);

}
2 Likes
3

Glad I was able to help. Well done on what you accomplished, sorry i couldn’t be more helpful. Can’t wait to see what else you do!

1 Like