How to consider only specifc geometry in a custom hlsl raytracer

BoardCogs
Wed, 18 Mar 2026 18:28:39 +0000
Programming & Scripting

So, for the past three months or so I’ve been trying to setup a raytracing global shader that tries to simulate rays bouncing off of surfaces. The point is that I want to ignore most geometry and I only want to focus on specific surfaces I have put in place.

I’ve followed this tutorial: Community Tutorial: How to Create a Custom Ray Tracing Shader as a Plugin and I made it work in UE 5.6, but now I’m stuck trying to filter out all geometry except the ones I need in one way or the other.

I’ve tried a bunch of approaches in many different ways, all fruitless up to this point:

1. Filtering out geometry in the TLAS before passing it to the shader as a parameter. I couldn’t make this work because the TLAS is derived directly from the FScene, and Unreal is very protective of this class and basically all others like FRayTracingScene and so on. It doesn’t allow to make a copy of these objects and trying to tamper with the ones used by every other engine process is obviously a recipe for disaster.

2. I tried looking for ways to filter by material in the shader. My geometry has a custom material, I thought maybe if I could get info about the material of a hit in the intersection attributes of any and closest hit I could just tell unreal to ignore everything that didn’t match. But I can’t seem to be able to access that information. Maybe it has to do with the fact that it’s a global shader, so variables with primitive data like material aren’t bound. i followed the tutorial I mentioned above, so the intersection attributes I get on the hit shaders are BuiltInTriangleIntersectionAttributes. As far as I know these don’t have any useful information, I tried searching in the source code and they don’t even seem to eist and I can’t find any information on them, is there a way to use another structure for the intersection attributes?

3. Last but not least, I tried looking for ways to filter geometry using the masks that are set when calling the hlsl TraceRay method. As an example, when calling:

TraceRay(
TLAS,
RayFlags,
RAY_TRACING_MASK_OPAQUE,
RAY_TRACING_SHADER_SLOT_MATERIAL,
RAY_TRACING_NUM_SHADER_SLOTS,
0,
Ray,
Payload
);

RAY_TRACING_MASK_OPAQUE is a bitmask telling the ray to only consider opaque materials. I investigated on the other masks defined by unreal and thought I could maybe define a custom one and give it to both my custom material and the TraceRay method to exclude everyting else. Again, it seems the bitmask for a material is computed internally and there’s no way to customize it without touching the source code (something I can’t really do, this is supposed to be a plugin), this means that this approach would be effective only if I decided to use a material with a specific bit mask for my geometries of interest and rays, and made sure that nothing else uses a material with the same bit on. This is also impossible in my case.

This is a summary of my many attempts, hours of skimming through the source code to find inspiration because I can’t seem to find any official material on this matter. It seems unbelievable to me that a custom shader wouldn’t be able to access information about the material it’s hitting during a raytracing pass, but I genuinely can’t find any ways to do it in Unreal.

If you have any info or suggestions I would appreciate every bit of help!

1 post - 1 participant