[Tutorial] Live 3D Content in your HUD or UI

My tutorial showing how to put live 3D content into a game’s HUD or UI is finally done.

I posted a preview of the tutorial a few weeks back, and thought that I’d have it done long before now. Some unexpected business travel put me off my pace, though, and I darn near forgot about it. Then Chance went and put the video in the Community Spotlight and that guilted me into getting off my butt and finishing it.

I apologize for the audio in a few spots. I was running a 102° fever through part of the recording and was sick as dog. That resulted in lots of coughs and other distracting noises getting captured in the audio. I’ve edited most of it out, but a few coughs, sniffles, and other noises made it through. You may notice a few dead spots in the audio where I removed the sounds.

It’s a very long video, clocking in at more than an hour. I went very much step-by-step through the entire process, setting up dynamic materials for the character, building the little color change spheres, building the proxy character, and putting that character into the HUD and syncing it with the player character. I even included a short section on minimizing the performance impact of the [FONT=Courier New]SceneCapture2Ds. More experienced people may wish to skip the first few parts, so I’ve put in a marker so you can skip forward to the live content in HUD part.

You can download the project right here.

So, here it is. I hope it’s helpful to some of you:

Just a note - Kitatus pointed out that this approach doesn’t yield a fully opaque character. If you want the character to be fully opaque, here’s an alternative material you can use:

Awesome work , been looking forward to this one!

Very nice, bookmarked it to sit down and go through when I get a chance. Nice work!

(From your other thread on this) Did you get a chance to look into this and find a way around it?

:

I really haven’t had a chance to look, but if I’m feeling better tonight, I’ll try and take a look at it.

Hope you get well soon mate. Awesome work on that tutorial btw.

Yeah that’s really awesome. great work!

:

Did a little playing around, and found a few things. Neither Temporal AA nor FXAA work on the scene capture as you probably already know. But, there are some ways to improve the look.

First, if you change the render target’s material from Translucent to Masked, you get slightly less noticeable aliasing around the edges of the character. You have to use the OpacityMask pin instead of the Opacity pin, and you still have to run it through a one-minus, but it does look nicer.

Another alternative, albeit a costly one if using a large render target, is to oversample. If you set the render target texture to 256x256 and then actually display it at 128x128, the downsample softens the edges in a way that’s nearly identical to anti-aliasing. Unfortunately, you need to go at least 150% to really see the results.

Still looking at options, but let me know if either of those is a viable option for you.

It’s long video, but it worths it to take more than hours to watch.
Awesome, Teff_lamarche! Your video is really helpful for me.
Thank you for your sharing!

Nice video.

The great things is that you are only focusing on the making the thing working but also give some hint to implement it the right way like activate a rendertarget only when it’s used.

I will give these a shot, appreciate you taking the time to look into it. :slight_smile:

Hey , I really like your solution to this problem. However I found a problem with it… You’re consistently running at around 40%-60% performance.
You don’t normally see this in PIE because it has it’s own hard coded settings and FPS smoothing that stays relatively low, but if you unbound and unsmooth FPS you can easily see the issue.

I modified your project to show the issue. I got rid of the face, and only kept the Body on tab press. I also have the FPS unbound in Standalone.

Observe the following results
Without Capture
Imgur

With Capture
Imgur

Full download:

1 Like

Thanks so much for the tutorial, it’s fantastic.
I am having a problem though. When I go outside the sky sphere to drop the camera and the actor, they both appear black, like they are unlighted. Also, when I go into the Render Target, the actor does not have the transparency deal covering it. Thus, I am unable to get rid of the background when it shows up on the viewport. I’m using 4.8 if that makes a difference. I can’t remember if when I started the project, which template I started in. I don’t know if that matters

This approach does not seem to work on UE 4.10.1. For some reason I get all white for the alpha channel of the render target. I also tried the project you’ve supplied and when I converted it to 4.10, it had the same problem. The current material returns opacity 1-x, and as x is always 1, the face/full body images are not visible at all. Tried removing the 1-x and feeding the x instead: the face with its background appears.

Really nice work mate! Good voice to listen to for tutorials as well. I’ll be picking apart this project for an inventory item preview, thanks!

Do we have any news regarding the SceneCapture2D Transparency issue or any workaround?

Seems that whole project doesn’t work at least at 4.9.2. There no UI pictures or any animation, only “player 1”. Is it real to repeat this giude, or now it’s just waste of time?

Upd: Got it. Then he removes background in the material with “1-x” node from alpha channel it works only in video. In fact we remove all picture. WHY?(

Thank you so much for this information! This helped me get an object to be interactively spinnable on my HUD.

I am having a couple issues, though.

  1. Any part of the 3D object that is 100% white on the HUD shows up as transparent. I cannot seem to figure out why this is happening. I have tried 1-x on the opacity and I tried the "If statement provided, neither one worked perfectly. Any idea how to get the background to disappear, but have the object show up completely opaque?

  2. Whenever I have the object rotating the refresh updates appear very wobbly. It looks like the model distorts slightly as it spins around. It seems like it would have something to do with the refresh rate or possibly anti-aliasing. Does anyone know of a fix for this?

If these are not possible right now, does anyone know of a good way to make a model viewer in Unreal? That is my ultimate goal here, to be able to click on an object, have it appear in the HUD widget if possible, and have it interactively rotatable. Unfortunately it also needs to look good. I’m not sure it can be done cleanly the way I have it setup right now. Any help is greatly appreciated. Thanks!

Unreal Engine 4.7.6

this doesn’t work in multiplayer setup. anyone know a solution?