Self-shadowing cannot be disabled, but one thing you may want look into is how addressed this very same issue in our GDC Kite Demo.
We used the Render to Texture Toolset to render out a depth texture for our tree imposter and then set everything up via the material so that the tree would have more accurate shadow casting and not appear like a billboard plane casting a shadow at these far distances.
The two additional pages for this tool you’ll want to look at are the Render 3D Imposter Sprites and the Render a Normalized Depth Map pages. You’ll want to render out the Imposter sprites so that when the camera is moving around these camera facing sprites will then shift between these imposters so that the view is accurate to the the location and the camera. The last page for the Normalized depth is so that you can capture accurate shadows for the sprites as they rotate. It can be a lot of setup, but it can be the difference between seeing a plane cast a shadow and something that blends more nicely for these billboards.
The Impostor stuff should be really useful for my lowest LOD and I have a few questions regarding that (see below).
However, the main problem I’m facing is with the full detail models + static lighting & shadows. Currently, I’m attempting a stylised art direction for my trees, using bent normals to make them look more volumetric. This works fine with shadows disabled, however, when shadow casting is enabled, the trees cast shadows onto themselves and the 2d cards I use for the fronds/leaves become very obvious:
Here you can see the difference between Shadows enabled (left) and disabled (right). With a Stationary directional light baked shadows are super sharp and a bit messy, and with a static directional light they are softer but the overall problem still remains.
Here’s a mock-up of what I’d imagine disabled self shadowing would look like:
The reason I can’t disable shadows completely is that it breaks the scene lighting - it just doesn’t look right when you’re walking through a forest where no shadows are being cast on the surrounding environment.
Maybe there’s a better way for me to approach this problem that I’m not seeing though?
Regarding the impostor stuff, I have a couple of quick questions:
1.) Do impostor sprites and normalized depth maps work well with baked static lighting?
2.) If I’m using static lighting for my trees and have an importer sprite as a LOD, is there anything I should do in regards to my lightmap UV’s on the main/impostor LOD to get it to light properly (since multiple lightmap UVs is not supported for foliage right now)
There may be some redundancy in my answers because I’m just answering as I go along. Sorry for that.
This works fine with shadows disabled, however, when shadow casting is enabled, the trees cast shadows onto themselves and the 2d cards I use for the fronds/leaves become very obvious:
I’ve never tried it with static lighting, but if you’re using a masked material for your setup for the imposter sprites I wouldn’t think it should matter here. You may need to check the option for Cast as masked shadow in the material settings though. In Kite Demo where we only used Dynamic lighting the PixelDepthOffset_Foliage node or use the VolumeShadowMap material function as a way to adjust the shadowing. It may be best to set this particular material not to cast shadows in the Static Mesh Editor under that Material Element since it can be faked in the material for casting on the actor.
For billboards you shouldn’t worry as much about completely accurate shadow casting for these because usually at that distance the player won’t be able to make out ever fine detail. If the tree mesh itself, and not the billboard, is casting a static shadow that will still be on the ground.
With a Stationary directional light baked shadows are super sharp and a bit messy, and with a static directional light they are softer but the overall problem still remains.
With your stationary Directional Light make sure that you’ve set a Dynamic Shadow Distance for the shadows. By default it is set to 0. At farther distances these will fade out anyhow and shift to the static shadow, so this shouldn’t be a concern for the billboards if they are being used with a static mesh as the higher LODs.
it just doesn’t look right when you’re walking through a forest where no shadows are being cast on the surrounding environment.
In Kite this was handled by using Distance Fields Shadows. You also have the option to enable Far Cascades for your directional light. This will use farther distances to have some cascaded shadows.
Regarding the impostor stuff, I have a couple of quick questions:
Do impostor sprites and normalized depth maps work well with baked static lighting?
If I’m using static lighting for my trees and have an importer sprite as a LOD, is there anything I should do in regards to my lightmap UV’s on the main/impostor LOD to get it to light properly (since multiple lightmap UVs is not supported for foliage right now)
Haven’t done any static lighting with them, but suspect it should work. Possibly some settings I’ve mentioned above that need to be enabled or options used.
No specific setup that I’m aware of for the imposter that needs to be setup for the UV. With the baked shadows from the mesh down to the billboard you’ll probably see some bake issues. I think this doesn’t come up that often is because foliage like trees that move and sway with the wind typically don’t get bake shadows because it doesn’t look natural. For the imposter sprite it may be better to just control everything via the material and use an unlit material that has these depth textures with the normalized shadowing. Use the emissive output to control the brightness of the material so that it’s not so blown out. I think it’s just a lot of finding ways to fake the effect or either not use the imposter sprites all together and have a significantly lower LOD rather than a billboard.
Thanks for your help and apologies for the delay in getting back to you. I wanted to try out your suggestions fully to make sure I understood them before I responded.
I tried what you suggested with depth maps on my foliage but I think there is some misunderstanding in what I am trying to do - most of your suggestions seem to be for improving shadows, where instead, I actually want to remove them:
Basically my issue is not about getting good quality self shadowing on my foliage, (the goal here isn’t realism, but stylization) the problem is that any self-shadowing on my foliage breaks the nice, volumetric, stylized shading from the bent vertex normals of my model:
All I want to be able to do is find a way to disable self-shadowing on my model, ideally via lightmass baked lighing (static/stationary) without disabling shadow casting entirely - Hence this mock-up image:
Hope this helps clarify things. I realised that disabling self-shadowing in Deferred Rendering is not easy (although they managed it in CryEngine somehow) but failing that, i’m guessing it must be possible for Lightmass baked lighting?
There’s no way to disable self-shadowing in UE4 so often you have to workaround this by using the texture plugged into a multiplier going to the emissive input. You can use a constant value to control the emissive nature so that you can override this self-shadowing so it’s not as intense.
If you have a sample project with one of these trees I that is stylized I could probably look and give you more direct feedback playing around with some settings and show you some examples.
Alternative methods would be to use an Unlit material and just control the diffuse in the emissive input as I described above.
Also looking at the material for the billboard and disabling shadow casting under the material option, but with that, you lose shadow casting from for this billboard for dynamic lights. But at far distances that shouldn’t matter. And if you’re using statically baked lighting anyway for these The LOD0 mesh would have it’s baked shadow on the ground and not the billboard. This also does not affect distance fields since the DF mesh is based on the LOD0 mesh and not the LODs down the hierarchy.
Anyway, if you have a sample you don’t mind sharing you can post here or send me privately on the forums and I can take a more in-depth look to give you feedback.
That’s sad to hear, I was holding out hope that it might still be possible in Lightmass since it’s baked and non-dynamic.
I’ve put together a sample project for you with some example trees (not pictured in this thread). One has two-sided polys with a single-sided material, the other is has single sided polys with a two-sided material.
This includes a test map which demonstrates different options turned on and off (shadow casting and mobility) - see attached file for project.
Thanks! I’ll take a look when I get time at the end of this week or early next week. Lots of stuff happening this week! (4.15 previews just started, Preparation for my training stream on lighting next week, documentation work, etc)
That’s sad to hear, I was holding out hope that it might still be possible in Lightmass since it’s baked and non-dynamic.
It’s still possible to use lightmass, just I think the best course here is to not use it on the billboard by disabling shadow casting for that part of the mesh. Any baked shadow for the LODs before the billboard should still be baked. That should work, in theory. I just need to see the setup and work it out.
Sorry, not yet. Busy week with releases and my time split between another team. I’ll try to get to it by tomorrow, but unfortunately, I can’t promise that I will be able to.
Hi all, I know this post is kind of old, but I wanted to chime in. To achieve the effect you’re looking for, you may want to look into vertex normal editing in Maya. This is how most artists achieve this “stylized tree” look that you’re after.
Since the question hasn’t really been answered. I believe the most simple and non-technical way to achieve this, is by having two objects. The tree itself, with" cast shadow" disabled, and another hidden shadow proxy, which would simply be a more optimised version of the mesh, thats set to invisible, but does cast a shadow.
Hope this helps some people.
How did you create double-sided polys? Everywhere on google keeps telling me the only way to do this is to either duplicate your faces or use the two-sided material, but your project files say otherwise.
I guess it’s a bit late to answer this thread but just in case if someone will be searching for this,
I’ve been searching for this same problem just recently and seems like there is still no option to turn off self shadowing in UE4.26 but there are some workarounds, you can try plugging in your Color Texture with a Multiplier into Emissive and Subsurface Color which will allow you to sort of control the shadow’s intensity, don’t forget to change your shading model to Subsurface!
(Many thanks to this post! Can a material cast shadows but not receive them - Rendering - Unreal Engine Forums)
Oh and don’t mind the PixelDepth Lerping into Opacity Mask, it’s a separate optimization technique which cuts off the alpha on great distances, here is where I’ve found it https://www.patreon.com/posts/25192376
(Triple thanks to all the authors!)
Also, bear in mind that those tricks can drastically lighten up your trees and make them much brighter!
RayTraced version is a bit darker in this case and actually looks pretty nice even without those Emissive, Subsurface and Opacity tricks from the start!
But those are just my small experiments, RayTracing is a different story for such stylized leaves, still trying out stuff myself, just hope this helps anybody!
Using translucent is quite expensive for tree shading. Check the SurfaceShadow in the FShadowTerms struct in ShadingModels.ush. This is the best solution.
