So after checking again with fresh eyes, there is couple of ways to do this and there is multiple things you should consider first before going into a direction.
- Since you are using a stencil on your shadow: even if we don’t highlight it in front when player gets into view it would be visible. Since object is visible and you just stencil cube.
- That means you need another shader pass on top if you want to hide player behind shadow. So the cube is always visible.
- There is couple of ways to do that as well but it really depends on camera since if camera is always one angle etc can be easier.
- For the object highlighted red to be behind or front of another object problem has also answer but either workaround or engine modification: To explain the problem : when custom stencil is rendered is only have one depth pass which is same for all ids. That means all stencil have same depth. We can have more depth passes for objects on scene view being rendered for custom stencils, however as said needs engine modification. Then we can have a method to compare which is behind which is front.
- Also depending on the game you making we can workaround the problem by camera depth as well if its a 2.5d game.
- Please do check lightening channels (there is 3) where you can have multiple different light sources and shadows depending on what kind of a visual look and feel you need. Maybe you don’t need fake shadows to do the thing you need (just assuming)
Let me know if you want an engine modification.
After telling all these here is a ugly workaround you can use for your game.
-post process
-you material mask is almost same basically
-here is how it looks
This is not the correct way of doing things however it works nicely on my computer with different camera angles distances etc.
I can take a look at whats happening deep in stencils also interms of depth
