Trying to implement textures for stylized shadows

I have a material that creates a basic hard shadow:


Which creates this:
HardShadow

I am trying to create a way to apply a sort of mask to create a texture at the edge of the shadow.

Here is a shoddily made mockup of what I am trying to achieve:

Does anyone know how I could approach this?

Yes you can. The shadow edge can be isolated by comparing the light vector and compared to the vertex normal using the dot product. This gives us a 1 to -1 range of values, where the edge is around 0. We can use this to create a mask that exists between a defined range, for example from 0.1 to -0.1.
We can use this mask to reveal a texture in a number of ways, such as using it as an alpha for a lerp.

I am not sure how I would go about applying a texture. Would you give me an example?

Just to clarify because maybe I wasn’t clear but I do want the texture (for example what I did with the zigzag mockup) to face the light direction.

The same way you would apply any texture to anything. UV maps or projections (such as tri-planar) It’s a more complicated if you want the textures directionality to change in response to the light’s angle, but you can alter coordinates mathematically based on the light vector.

In this video, around the 18 minute mark, I show an example of changing a 3D texture coordinate system based on a light vector to fake dynamic caustics. Although I think there might be a pre made node that rebases the Z direction of a coordinate system to face an arbitrary vector, I do so manually there.

The issue that I face is that I don’t know how to rotate the texture without angular distortion.

I got this texture right here

If I were to rotate the UV I will just get this:


Which is not what I want of course.

The world space projection I show in the linked video does not cause distortion when rotated.

I see, I am running into the problem that I can’t figure out how to make use of that UVW.

Let’s say I isolate the area before the edge of the shadow like so:

I have not worked with volume textures before and I can’t find much information on how to utilize them in a way that isn’t fog or clouds so I am having a lot of trouble trying to make sense of the UVW information and how that would create the desired result

A UVW is basically just a world position. And a world position is just a coordinate. That means you can use it for 2D world aligned (aka tri-planar) texture coordinates just as easily as a volume texture.

You can also simply generate coordinates with your dot product mask too. Remember we have a value going from -1 to 1 along the mesh surface from the dot product. A coordinate system should run from 0-1 so you need to mathematically manipulate the mask’s band into the 0-1 range. Let’s call that your Y coordinates.

The X coordinate is a bit harder, because you need to find a way to wrap it around the mesh instead of along it. You could probably again use a dot product but from a vector that is orthogonal to your original light vector. This would give you another -1 to 1 gradient, but wrapping around the sphere. If you add 1 and divide by 2 you now have a coordinate system that wraps around the mesh.

There are a lot of different ways to project a coordinate system. What you’re doing is a bit atypical so you’ll need to think outside the box when generating them. Ask yourself how you can create a 0-1 range along the shadows edge (this is technically known as a shadow terminator), and a 0-1 range that wraps around the mesh for your second coordinate.

You’ll need to understand how to generate/project coordinates systems very comfortably to do this properly. There are probably several approaches that can work and each will probably have some pros and cons.

Thanks for the information, I’ll try to get some more knowledge before I proceed.

I just did this exact thing the other day. Here’s my solution: (It is slightly different if you’re using a color vs a texture, but you can see where I did both. Those colors are just there for testing. Also I did my sunlight vector from a material parameter collection vector. I recommend that so you can use it for other stuff and only calculate it once in the level blueprint.) You can also do something similar with smoothstep to get a softer edge (it works I tried it) but I liked this better. The most relevant part for you is the “Adds in Middle Color” part. I put in those node dots for you for where you swap in the stuff you want for facing, away, and in between.