Gallery PsyGan | Immersive AI Projection Mapping with Unreal Engine 5

Community Showcase
, ,
1

This psychedelic gallery pairs AI generated art with #UnrealEngine5 and virtual reality to produce a unique exploration of dimension and visual style.

Here is the full gameplay runthrough: Gallery PsyGan | AI Gameplay Demo 4K with Unreal Engine 5 - YouTube

Feel free to give it a play here: Gallery PsyGan by ThoseSixFaces

I have scattered 40 unique hotspot throughout the gallery for players to discover. Each hotspot takes the player down a surreal mindbending reality. The goal behind this project was to turn the unique style of AI generated art into a playable realtime experience with 6 degrees of freedom where players are able to freely roam around the volume.

My pipeline for creating Gallery PsyGan involves:

  • Blocking out the gallery environment and scattering hotspot locations to fill the volume.
  • Creating a batch renderer that captures a 360 degree bespoke high-contrast ambient occlusion post processing pass at each location
  • Feeding each photo through VQGAN+CLIP (a Text-to-Image Model) with unique prompts for 1000 iterations. The AO pass helps guide the AI towards breaking up features in the final output where intersecting geometry appears. This ensures there is still some coherence of the original dimensionality.
  • AI upscaling the artwork with Topaz Gigapixel for a high definition experience.
  • Projecting the AI art at the locations their source image was captured by developing and using a custom spherical projection decal shader in Unreal Engine.
  • Interpolating the decal intensity when the player is in weighted proximity of each hotspot.

Unreal Engine has been a huge help in speeding up the creation process for custom pipelines like these.

5 Likes
2

Here is the breakdown condensed into a single image for your viewing pleasure:

breakdown_image
breakdown_image1797×2376 318 KB

1 Like
3

Greetings @Lozemo76 !

Thank you for sharing your project with us! I’ve seen people create some really cool, trippy images and live art with AI-generated art. I’ve never seen anyone apply it to UE! What a novel idea! The possibilities are endless!

Did it take you long to figure out how to integrate the AI-generated art into the engine? How did you come across the idea to do so?

2 Likes
4

Man, this is something UNIQUE and REVOLUTIONARY! Congratulations for trying some new and AMAZING things with UE!

1 Like
5

Thank you!

I had the idea in my head a couple years. I tried building a prototype in Unity but the shaders were a bit painful to work with. I found Unreal to be a one stop shop for everything I needed from capturing the 360 photos in batch, to reapplying them onto an untextured world using a custom post processing pipeline.

Once I setup my hotspot zone/projection blueprint, it was relatively quick to capture photos from all hotspots, process through AI and feed them back into a prameter on the blueprint. I had a nice enum that I could switch between ‘Capture Photo’ and ‘Project Photo’ on the individual hotspots.

2 Likes
6

Just love it. Looking at this gives me some flashbacks into the 80´s where i hade somtimes the same feelings. The only thing i miss is some cool music. :sunglasses:

1 Like
7

Ah yeah, some old 80’s techno music would fit perfectly :grin: