I’m making a 3D fog of war system and to make it I use a custom render pass as the mask for post-process material as to which areas should be hidden. It works basically as a shadow map. It works, except for high resolutions.
Here’s how it’s supposed to look and how it indeed looks in the viewport. Viewport’s size here is something random like 1600x600.
The problems appear if I resize viewport to be larger or run in standalone mode. Here’s the standalone mode at fullscreen on my 1920x1084 monitor.
But the problems get even worse if I run the game from a packaged build. On the same 1920x1084 monitor:
I have extensively tested the materials I’m using and the problem definitely definitely stems from sampling my custom render pass. Here’s the post-process material that applies the fog of war to the scene using my custom render pass as a mask.
For completion’s sake here’s what I write into the render pass, by sampling a depth cubemap to make a shadow map:
In short: how do I get rid of the banding and bleeding of the post-process effect caused by sampling a custom render pass.