Dawn of an Old Age interactive dance performance

Dawn of an Old Age is a live dance performance that uses interactive visuals, built in Unreal Engine. The visuals are projected onto the dancer and a white wall, using a very bright projector. Using different motion tracking techniques, I make it appear as if the dancer is inside and interacting with the 3d worlds being projected. The video speaks for itself. I used UE4 in two sections of the piece:

First section (0:00)

This section uses a Theta V 360 livestreaming camera, streaming over a private network. The dual paraboloid frames from the camera are mapped to the inside of a UV sphere using a custom material that stitches the two halves of the camera lens together. The dancer is “inside” of this sphere for most of the performance, and her turning allows us to see different views of the live-streamed video. This material was also built with the ability to “burn up” with a electrical-ish fire (towards the end). Also towards the end, I use instanced meshes to create hundreds (thousands? I don’t remember) of similar spheres that also have the live video mapped inside them.

Second section (18:00)

This section relies heavily on Niagara and GPU particles, including a custom Niagara Module Script that can emit particles according to a mask. Normally, if you try to emit particles from a texture mask, you have to do it in a grid, and kill the particle if-and-only-if the grid location is outside of the mask. This is inefficient. The custom module allows you to pump out all the particles only where the mask exists, resulting in a much more efficient use of particles. Instead of only a percentage of the particles living (with the majority being killed), 100% of the particles live and show up only where the mask allows them. I also use Ultra Dynamic Sky and Aquatic Surface plugins for the main elements in the scene.

As a consequence of building this piece, I’ve created a fairly robust framework, that anybody can use, for creating more live performances like this. The framework handles things like section transitions, sound checks, live control of different elements, intuitive camera control, and motion tracking.

Congratulations for your work,
I have a question about motion tracking. It was done entirely in Unreal or sent to other software via spout or someting.

Thank you for your kind words. The motion tracking was done using a thermal camera, OpenCV and some custom C++ code. If you are interested in making this kind of work, I will be producing some documentation that explains how I accomplished the project with Unreal.

If anyone else is interested as well, please bump this thread so I can get a sense of who the audience is :slight_smile:

Interest intensifies! I’ve been playing with a similar concept using Kinect - latency was an issue for mapping onto moving bodies though. BUMP!

Absolutely… latency is the killer for these kinds of projects. I’ve found that around 90fps device is ideal. A 60fps device is usable (used in this project), but even that is very noticeable.

Wow, that’s sick! I’m also interested in using Unreal for interactive dance performances - so yeah, BUMP!