AR implemented with OpenCV webcam capture drawn to dynamic texture

Hi All,

I’ve implemented a proof-of-concept of Augmented Reality in Unreal Engine using OpenCV webcam capture and drawing to a dynamic texture.

You can see a screen capture of the example here:

You can also see the source code on GitHub here:

Would welcome any feedback! Now that the basic foundation is in place I will look to implement additional AR functionality like detecting markers.

will have to see how easy is to play with it.
I have never programmed with C++.

Btw,you can look how the video device in approach was build in touch088.
Maybe you can make "simple " blueprint " solution that contain the same parameters layout as here.

Very cool :slight_smile: The webcam might need to be warped to fit the Oculus FOV if you have it mounted, but otherwise a very solid PoC!

Wonder what the UI styles work effectively in AR, with Vive it seems we will at the very least have our hands as inputs.


Yes, the Vive is a very cool input concept for games where of course you want the best controllers, but I’m wondering if it would really be mass-market. Then again Nintendo sold a ton of Wii consoles, so who knows?

Personally I would like to try mounting the Leap on a Samsung Gear VR to show how you could have a self-contained HMD without need of additional controllers in a VR application (especially since the Samsung Gear VR has a built-in touchpad anyway). So if the object was close to you you could interact with it with your hands, and if it was farther away you could just use the touchpad with the raytrace input concept in my demo.

I think the mass-market VR adoption will be with the mobile devices like Samsung Gear VR or Google Cardboard based devices, and the market leaders will ultimately be Facebook/Samsung and Google. Google is certainly formidable competition for Oculus / Gear VR, but then again Apple is holding its own with the iPhone vs. Android phones. So Google isn’t invincible but Oculus and Samsung will definitely have to offer very compelling products along with Apple-level marketing.

Hi i have no experience with unreal engine i want to use your code to add PlayStation 4 Camera device support under osx. I don’t understand how to use your information included in the readme, is there a tutorial or step by step howto for unexperienced unreal engine user like me?

I have published a userland driver for PlayStation 4 Camera here:

I can do the PlayStation 4 Camera interface class and add custom video source for it but i must first learn the basic about unreal to make it. Can you help me?

Hi Imalave - this video on a texture business looks great! Do you know how easy it would be to use a source like a Blackmagic capture card instead of a webcam? I’m trying to make a kind of virtual studio with keyed people rendered on to quads.

I’m going to take a look at your source when i get a moment (i’m not really a programmer though) and this OpenCC sdk which looks very interesting.

Thanks for your input here.


Sigh - I’m such a newb - I couldn’t get it to compile. I’m using the 4.72 build and i’ve tried both the OpenCV 3.0 beta (which seems to do best) and the 2.4.10 version.

I’ve added paths to the ‘Include Directories’ bit in VC but get compile errors like -
Error 9 error C4003: not enough actual parameters for macro ‘check’

Any ideas, oh ye altruistic UE companions?


Hi Dannington,

Sorry, I don’t know what a Black Magic capture card is. Can it be detected in OpenCV as a webcam video source using “VideoCapture(<camera index>)”?

I’ve never seen any compile error related to a Macro Check, but here is what I have in my build.cs file using OpenCV 2.4.10

   if (Target.Platform == UnrealTargetPlatform.Win64)
        PublicAdditionalLibraries.AddRange(new string] { "C:/LeapSDK/lib/x64/Leap.lib", 
            "C:/aruco1.2.5-win_32bits_msvc10/lib/aruco125.lib" });
        PublicIncludePaths.AddRange(new string] { "C:/LeapSDK/include", "C:/ovrvisionsdk/include", "C:/opencv/build/include", "C:/aruco1.2.5-win_32bits_msvc10/include" });

Can somebody link to me a proper tutorial that explains the following:

  • how can I add c++ classes to visual studio and make them visible in Unreal editor
  • how can I add the files from this topic (the readme assumes I already know the first point) to a current unreal-vs working project?
  • how and in how many ways can I configure opencv to work with unreal project?

What I did was to copy the source files in the project directory. Visual studio does not see them. So I added a new folder inside visual studio (called filter) and then dragged the files inside that folder. Now it can see it, but editor do not. Then the readme asks me to create a blueprint as a subclass of VideoDisplaySurface. How am I supposed to do that, since editor cannot see the classes I just added?
The steps to do are not clear to me.
I then closed the editor and tried to compile. And I have a lot of LINKER errors (yes I renamed the .h correctly even though it was not written in the readme).
And now, when I try to open editor again it asks me to recreate the .dll . And whether I say no or yes (compile fails) editor won’t open. So now I am stuck.

I am using latest version of the engine (it is the one Epic Launcher installs) and I was able to import the three classes in the project. As far as I understood only classes that derive from UE classes can be viewed in the editor.
Still I have linking issues. In particular after adding the class AVideoDisplaySurface. I have updated the constructor for the new UE version, but these linking errors I still don’t understand where they come from. Some errors on Texture2D are displayed I will look into it.

ok so when I add OpenCVVideoSource or VideoDIsplaySurface gives me the same dllimport errors. Probably related to opencv. I get sometimes that visual studio compile while UE does not. I hate this.
Is there any tutorial to how configure opencv with unreal engine? Does it have to be a plugin? Or a lib? Personally I used unreal tutorial on how to implement third party static lib. Don’t know other ways…

EDIT3: I got it to compile from Unreal editor, but not from VS by creating a project from scratch. Watch out UE deletes your configurations if you run refresh vs project.
Anyway I cannot create the BLUEPRINT to derive from VideoDisplaySurface… that is because UE does not see VideoDisplaySurface at all!! As any of the classes I put into vs.
Do you mean I have ALWAYS to create classes from within UE??? This is dumb

Hi fakkoweb,

Sorry about your issues compiling! Ideally these classes would be packaged in a plugin so that it would be much easier to integrate, but I haven’t gotten around to it yet since I’ve never created a plugin.

So yes, what you can do is just create a class called VideoDisplaySurface from UE by doing “Add Code To Project”, and then copying the contents of the github file to the new file that is created. Not ideal and I’m sure there is another way, but this way will definitely work.

As far as making files visible in Visual Studio if they are already on the filesystem, you should be able to right click on the project in Visual Studio and select “Add Files”.

Let me know if you encounter any more issues. I guess you already figured out the OpenCV integration, but I just installed the SDK on my computer and then I just add the paths to my lib and include paths as I show in the post higher up in this thread.

Yes I got it compiling, but adding a new class already in the file system to VS does not show up in UE. This makes me sick.

Anyway I am still reading UE documentation. As far as I got, blueprint and c++ classes are the same, they only change in functionality since the first is easier to use (through editor).
What I am trying to understand right now is where the BeginPlay() code should go.

I still am a newbie in UE but what I got from “AVideoDisplaySurface* BackgroundVideoDisplaySurface = (AVideoDisplaySurface*)BackgroundVideoSurface->ChildActor;” is that I should have some actor (?) that has access to the child of some BackgroundVideoSurface (?) which happens to be an AVideoDisplaySurface. Then to it I attach a VideoSource and this is clear.

I apologize probably this should be pretty clear except from who has still to comprehend the mechanics of UE (but I have some experience in Unity)

Hi fakkoweb,

So sorry - I realized that I have some code in my Character class constructor that I did not include in the README instructions:

static ConstructorHelpers::FObjectFinder&lt;UBlueprint&gt; VideoSurfaceBlueprint(TEXT("Blueprint'/Game/Blueprints/VideoDisplaySurfaceBlueprint.VideoDisplaySurfaceBlueprint'"));
if (VideoSurfaceBlueprint.Object){
	VideoSurfaceBlueprintClass = (UClass*)VideoSurfaceBlueprint.Object-&gt;GeneratedClass;

// Create a mesh component that will be used when being viewed from a '1st person' view (when controlling this pawn)
BackgroundVideoSurface = ObjectInitializer.CreateDefaultSubobject&lt;UChildActorComponent&gt;(this, TEXT("BackgroundVideoSurface"));
BackgroundVideoSurface-&gt;AttachParent = FirstPersonCameraComponent;
BackgroundVideoSurface-&gt;ChildActorClass = VideoSurfaceBlueprintClass;

I am an Unreal newbie as well (about 3 months experience), but what this code is doing is creating a ChildActor component that is attached to your character and will move with your character like a rigid body. You give it the class to instantiate (in this case the class for VideoDisplaySurfaceBlueprint). So what happens is this will create an instance of the blueprint that is always in front of the character and taking up the entire field of view, so it essentially becomes a background video. In my example I put this surface about 5 meters away so that you can draw objects in Unreal in front of this background video.

Sorry for the omission - I will add these instructions to the as well.

Configuration Question

Hi Imalave,

Thanks for the awesome example!

I noticed you were using the release versions of the opencv libs. Which visual studio build configuration do you typically use when compiling?

I have to use the debug versions (i.e. “opencv_<module>2410d.lib”) to get it to compile with the Development Editor Win64 config on Unreal Engine 4.7.2 even though the docs state that the “Development configuration is equivalent to Release.”

On a related subject, are you able to get your opencv pdb’s to load successfully? I’m getting an exception in cv::imread() and cv::imwrite() but I’m unable to read the exception message because opencv_core2410.pdb is not loading properly even though the VS debugger is finding it. Visual Studio 2013 is stating that “PDB does not match image” (in Debug > Windows > Modules > right-click opencv_core2410.dll > Symbol Load Information). I’m using the libraries, dlls, and pdbs from the vc12 folder. Just thought maybe you’ve encountered something similar.

Thanks again!

I think I have this nearly working. I’m still running into an issue where

Is NULL. Which I don’t understand because I am successfully settings ChildActorClass to the blueprint (confirmed in debug mode)

Any idea what’s missing?

Hi here are some dumb questions, but i am juste started on unreal engine and i did not find any answers for these :

  • how do you do that : “Create a Blueprint using VideoDisplaySurface as the base class” ?

But in any case very nice job man !

Hi bwiklund,

Is it in your Character’s BeginPlay() function where you have this line of code?:

AVideoDisplaySurface* BackgroundVideoDisplaySurface = (AVideoDisplaySurface*)BackgroundVideoSurface-&gt;ChildActor;

It needs to be in the BeginPlay() function to ensure that the Character has already been initialized.

Hi c9iner,

Hmm…I almost always the Development Editor in VC++ and I haven’t had issues. It might be because I “cheated” and copied the OpenCV DLLs that I needed directly to the Binaries/Win64. As for the PDBs I didn’t do anything with them. I only copied the DLLs manually to my Binaries/Win64 folder, and then I configured my PublicAdditionalLibraries as I described in my post above.

Unfortunately since I am an Unreal Engine newbie I am probably not setting up my builds and packages in the best way. For example, all of this code I’m building should really be packaged into an easy-to-use UE4 plugin but I just haven’t gotten around to it yet.

Hi Razgort,

Sorry I don’t have screenshots, but basically you go to the Content Browser window and then select “Add New > Blueprint Class”. That brings up a dialog where you can select the parent class. At the bottom of that Window you can browse “All Classes” to find it, or enter the name into the search box that is there.

Yeah, it’s in there. I was able to make a workaround by not using a blueprint, and just using #StaticClass of AVideoDisplaySurface. Still don’t understand what went wrong with initializing the blueprint though.

Hi, I’m also very new at Unreal. What is OculusARPOC.h? I can’t get to compile, because it’s a missing reference and I couldn’t find anything with google