Making holes in geometry (material that doesn't writes to color buffer)

I’m trying to create material that isn’t writing to color buffer, but only to depth buffer. In order to do that in Unity, I simply had to specify those two lines in the shader code:

Tags{ "RenderType" = "Opaque" "Queue" = "Background-10" }
ColorMask 0

Also, how can I specify rendering order if I have multiple cameras? Again, in Unity I just had to change Camera.depth property.

Bump… How do I only write to depth buffer, and not color buffer?

That won’t be as easy as in unity :stuck_out_tongue:

You can use a masked or translucent material of course but that will also not write the masked area in the depth buffer.

If you only need the mesh’s depth then you can set the mesh to write only to the custom depth buffer (it’s a secondary depth buffer which can then be accessed in a material).
If you really need a hole but also have the depth buffer then you can either mess around with the engine source and implement that kind of feature (that would be a major pain in the butt) or use a duplicate mesh and set it to write to custom depth (not a great workaround but it’s pretty much all you can do).

Alright, let’s suppose that I’m writing only to the custom depth buffer. How can I render another camera’s frustum in that place? I can’t find any documentation on overlaying multiple cameras in UE4…

That can potentially be done via a SceneCapture2D component but it is expensive, there is no easy or performant way to overlay multiple cameras in UE4.
UE4 uses the deferred shading path (instead of forward) and the overhead for multiple cameras can be huge.

May I ask what is your end goal?

Well, Unity has RenderTexture instead of SceneCapture2D, but as you mentioned, it has huge overhead (you are basically rendering everything twice with same resolution). I found that I can take advantage of portal rendering in order to achieve same effect, with almost no impact on performance.

You can check here for OpenGL example of such setup;

Antichamber is based on UDK or UE, and is probably doing something similar to make non-euclidean levels. While that is not my end goal, I need similar setup.

TK-Master, thank you for your help so far! :slight_smile:

EDIT: To describe in other way, I’m looking to render first camera’s frustrum, except in place of some object, and fill that space with another camera’s view. That should cause any performance issues, but I’m not sure how easily can it be done in UE4.

I have seen a few attempts for portals in UE4.
Most notably by @GalaxyMan2015 and more recently by @TheJamsh
But I believe all of them were via SceneCapture2D… I could be wrong though, you might wanna send them a pm.

I have contacted @GalaxyMan2015. Apparently, there is no way to do this without tinkering with engine code (which he did, but lost code unfortunately). It’s kind of disappointing that Unity has more flexible rendering system :confused:

TK-Master was correct, my/that solution uses a Scene Capture based system. The C++ is done so that it works as a plugin rather than an engine modification. Scene Capture isn’t really any different than a ‘camera’ though really, you just have more control over what it renders or processes. Anti Aliasing is a problem unfortunately, as you might have seen in my solution - and tweaking the parameters for the perfect teleport is a bit fiddly too.

I wouldn’t say Unity is more flexible, it’s probably just more easily exposed. With full source code to the renderer there’s no limit to what you can do if you’ve got them mad programming chops. I haven’t looked at it personally, but stencil buffer or not I’m pretty sure you still have to render a full set of GBuffers for each frustrum. The stencil part is just masking between them?

I’m not remotely surprised that modifying the renderer requires engine source to be honest. I imagine that supporting something like this natively adds a lot of bulk to the renderer and slows things down. UE has a lot of bells and whistles, it’s important to keep it as optimized as possible. Shame GalaxyMan lost the code to his solution though, hope he works it all out again.

I would post source code for my solution (that was the original plan), but I fell into an IP dispute over it so I’m afraid until that’s over it’s gotta stay on my hard drive only :confused:

EDIT: Forget what I said about rendering a full-size GBuffer for each frustrum - you only render the stencilled part of course, hence the speed improvement. Post-Processing might be an issue when jumping between different environment etc, but it might solve the anti-aliasing issue too.

Still, most of the code that does the heavy-lifting for this should already be in the renderer for things like Custom Depth. I’m fairly sure it’s just a 256-channel stencil buffer really.

EDIT #2: Some more info: Stencil buffer - Rendering - Unreal Engine Forums
@DanielWright or @RyanB are probably the best guys to answer these kinds of questions, though.

To do realistic portal with Scene Capture to texture, you will need to render two scenes at full resolution - thus the performance overhead.

Another issue is that texture is probably rendered from one frame before. Have you managed to solve that lagging issue in your implementation, @TheJamsh?

I have no problem editing engine code, but since I just started learning UE4, codebase seems daunting for now.

Yeah I can understand that!

I’m not sure if it gets around it or not since I haven’t noticed any lag, but I don’t let the scene capture update itself, instead I force it to update in the code manually. Otherwise it runs at it’s own framerate independant of the rendering thread.I calcualte the transforms of the scene capture first, then tell it to update the render target.

The render target is fed into a material as a texture parameter, along with the clipping plane of the frustrum as vector parameters. Using that I then calculate the UV distortion of the texture so that it draws properly.

APP_Portal::APP_Portal(const FObjectInitializer& OI) : Super(OI)
	CaptureComponent = OI.CreateDefaultSubobject<USceneCaptureComponent2D>(this, TEXT("Capture Component"));
	CaptureComponent->bHiddenInGame = true;
	CaptureComponent->bCaptureEveryFrame = false; // We do captures manually - can be overridden if required though.

 	PrimaryActorTick.bCanEverTick = true;
	PrimaryActorTick.TickGroup = ETickingGroup::TG_PostUpdateWork;