Augmented Unreality - AR plugin for UE4

I would like to present my augmented reality plugin:




  • Display camera stream in-game
  • Augmented reality, camera pose tracking with ArUco markers
  • Camera calibration


  • Editable marker configurations
  • Tracking independent boards
  • Switching between video sources
  • New video sources: file, stream
  • Build for UE 4.13.1


  • Android, Linux
  • ChArUco boards (chessboard for better accuracy)
  • Build for UE 4.15


  • Build for UE 4.16

The code is open source (Apache license) and may also serve an educational purpose for the following topics:

  • Integration of OpenCV and UE4
  • Multi-threading in UE4
  • ArUco marker tracking
  • OpenCV camera calibration
  • UE4 dynamic textures

I hope the community will find the project useful.

Hi Adynathos! Thank you for this piece of the code… I have sent you PM if you can respond… Thank you.

Ooh, nice! Does it come with pre-built plugins for Windows/iOS/Android (don’t see binaries on Github) ?

AR using video from drone seems to be extremely useful for all kind of architectural projects! (whether to show real-time presentation or render out offline video from UE4)

The binaries are here:
The link is given in the repo’s README, maybe I should also add it to github’s releases.

There are full pre-built binaries for Win64 and also third-party shared libs for Linux (although right now it crashes on Linux).

I see (downloaded). So no Android support yet. Do you have any plans for it by chance ?

It would indeed be fun to run this on Android. To make it work there, we would need to:

  • build OpenCV with aruco module from OpenCV-contrib, OpenCV seems to work on Android
  • build the small OpenCVWrapper shared library
  • add those shared libraries to Unreal’s Build.cs
    Unfortunately I don’t have experience with building UE projects for Android (I only worked with simple Java-based Android programs). Do you know if adding external libraries in UE works with Android?

Hi, Thanks for your plugin,
It will be great if you can add support for Android and iOS

I tried to compile opencv for unreal myself and failed miserably :<
Hey, did You tried any experiments with visual odometry? Is it possible to implement it using Your plugin?

It would be cool if 4.12.x binaries were provided (and 4.13.x) :slight_smile:

Still no Android support?

Any plans for implementing support for Gear VR too ?

What are the current limitations for markerless detection using opencv? We know it works really well in other applications. But what about unreal4?

Thanks a lot for your great work, ZkarmaKun.
However, I didn’t get the idea of applying this masterpiece.
I’ve created an new actor and created a new blueprint based on it, and even created a blueprint of the gamemode, both of the two eventgraphs are exactly the same like yours, but what appeared on my Android device were just a floor and a cube.
Would you please tell me whether I missed some important process?
BTW sorry for my poor English, I’m Chinese, if my words is not that appropriate, please don’t take offense.><

Sorry for the lack of reply - it seems I did not receive the notification.
There are many features I have added since the first release: spatial marker configurations, switching video sources, multiple marker tracking.
I will try to make a release for the new features soon.

New version released! - please see in original post.

OpenCV has implementations of SURF and SIFT features, I think I have even seen someone implement positioning on OpenCV. So this should be possible.
One difficulty though is determining the initial pose of the camera.

Height and rotation of the camera can be set arbitraly, as well as the position in virtual space
or im missing the point completely.

Of course, but imagine you want the AR objects to be on your table. With markers, you place them on the table and we know that is the Z=0 plane for example.
With visual SLAM you do not know where the table is. Also there is always some drift and no good way to correct it.

Yea, especially with monocular VO, binocular has the ability to get some deph information, but it seems that the only valid approach are RGBD cameras like kinect (obviously too heavy for headmounting) and realsense. Im waiting eagerly for intel tango phones for that matter.
Well there is much less drift with SLAM with loop closure, and i’ve seen some applications of Large Scale SLAM with negigible drift. Here is an example

Easier than performing full SLAM is to use a known object as marker, the object does not have to be a fiducial marker. There is an implementation in OpenCV.


I’m just wondering if this is compatible with the current new version of UE 4.14.

Thanks for the answer

It should be.
But if you want to avoid rebuilding it, you can download this current development version which was built with engine 4.14:


I would like to know if I can achieve the same result with this plugin what they showed at GDDC with the black bird. Multiple trackers on one object for better / stable tracking from multiple angels. Video:
If so how can I do this? Thank you :slight_smile: