Augmented Unreality - AR plugin for UE4

Community Showcase Game Development
1

I would like to present my augmented reality plugin:

Downloads

Documentation

Features:**
1.0.0**

  • Display camera stream in-game
  • Augmented reality, camera pose tracking with ArUco markers
  • Camera calibration

1.1.0

  • Editable marker configurations
  • Tracking independent boards
  • Switching between video sources
  • New video sources: file, stream
  • Build for UE 4.13.1

1.2.0

  • Android, Linux
  • ChArUco boards (chessboard for better accuracy)
  • Build for UE 4.15

1.2.02

  • Build for UE 4.16

The code is open source (Apache license) and may also serve an educational purpose for the following topics:

  • Integration of OpenCV and UE4
  • Multi-threading in UE4
  • ArUco marker tracking
  • OpenCV camera calibration
  • UE4 dynamic textures

I hope the community will find the project useful.

https://polybox.ethz.ch/index.php/s/Kc4hJHLdvjARvnJ/download

https://polybox.ethz.ch/index.php/s/H0GGOXE5XZNL9Ci/download

https://polybox.ethz.ch/index.php/s/31CTBjbqRJYoiqT/download

2

Hi ! Thank you for this piece of the code… I have sent you PM if you can respond… Thank you.

3

Ooh, nice! Does it come with pre-built plugins for Windows/iOS/Android (don’t see binaries on Github) ?

AR using video from drone seems to be extremely useful for all kind of architectural projects! (whether to show real-time presentation or render out offline video from UE4)

4

The binaries are here: AugmentedUnrealityProject_1_00.zip
The link is given in the repo’s README, maybe I should also add it to github’s releases.

There are full pre-built binaries for Win64 and also third-party shared libs for Linux (although right now it crashes on Linux).

5

I see (downloaded). So no Android support yet. Do you have any plans for it by chance ?

6

It would indeed be fun to run this on Android. To make it work there, we would need to:

  • build OpenCV with aruco module from OpenCV-contrib, OpenCV seems to work on Android
  • build the small OpenCVWrapper shared library
  • add those shared libraries to Unreal’s Build.cs
    Unfortunately I don’t have experience with building UE projects for Android (I only worked with simple Java-based Android programs). Do you know if adding external libraries in UE works with Android?
7

Hi, Thanks for your plugin,
It will be great if you can add support for Android and iOS
Regards

8

BEAUTIFULL WORK!
I tried to compile opencv for unreal myself and failed miserably :<
Hey, did You tried any experiments with visual odometry? Is it possible to implement it using Your plugin?

9

It would be cool if 4.12.x binaries were provided (and 4.13.x) :slight_smile:

Still no Android support?

Any plans for implementing support for Gear VR too ?

10

What are the current limitations for markerless detection using opencv? We know it works really well in other applications. But what about unreal4?

12

Sorry for the lack of reply - it seems I did not receive the notification.
There are many features I have added since the first release: spatial marker configurations, switching video sources, multiple marker tracking.
I will try to make a release for the new features soon.

13

New version released! - please see in original post.

OpenCV has implementations of SURF and SIFT features, I think I have even seen someone implement positioning on OpenCV. So this should be possible.
One difficulty though is determining the initial pose of the camera.

14

Height and rotation of the camera can be set arbitraly, as well as the position in virtual space
or im missing the point completely.

15

Of course, but imagine you want the AR objects to be on your table. With markers, you place them on the table and we know that is the Z=0 plane for example.
With visual SLAM you do not know where the table is. Also there is always some drift and no good way to correct it.

16

Yea, especially with monocular VO, binocular has the ability to get some deph information, but it seems that the only valid approach are RGBD cameras like kinect (obviously too heavy for headmounting) and realsense. Im waiting eagerly for intel tango phones for that matter.
Well there is much less drift with SLAM with loop closure, and i’ve seen some applications of Large Scale SLAM with negigible drift. Here is an example Large-Scale Direct SLAM with Stereo Cameras (IROS '15) - YouTube.

17

Easier than performing full SLAM is to use a known object as marker, the object does not have to be a fiducial marker. There is an implementation in OpenCV.

18

Hey,

I’m just wondering if this is compatible with the current new version of UE 4.14.

Thanks for the answer
Rokew

19

It should be.
But if you want to avoid rebuilding it, you can download this current development version which was built with engine 4.14:

20

Hello,

I would like to know if I can achieve the same result with this plugin what they showed at GDDC with the black bird. Multiple trackers on one object for better / stable tracking from multiple angels. Video: https://www.youtube.com/watch?v=VJEoY1JT71c
If so how can I do this? Thank you :slight_smile:

21

The plugin uses very similar technology to the one shown on GDC (they also said they use OpenCV but Chilitags not ArUco).

A multiple marker configuration can be used - markers are placed in the blueprint editor (as actor components). In the GDC demo, they put the markers on the car then scanned them, in this plugin you need to know the position of the markers.

If you are interested in trying the plugin, please let me know. so I can package the current version for you (the last published one is far behind current state of the plugin)