How objects ignore the plane in front of them

Hello guys!

We are making an interactive game for a multi-touch screen. The game has characters and a green world behind them.

We want to implement the mechanics of coloring (the appearance of color from BW) characters and the world around them simply by swiping.

The characters are raycasted, we change their UV texture from B&W to color. There is also an autofill if 80% of the character’s texture is painted.

Now we don’t understand how to correctly implement decorating the rest of the world. Doing the same raycast seems to be inappropriate, so we decided to put a plane in front of the camera that decorates the texture that it sees in front of it. The solution works well, but the problem is that if the character is fully colored, and the plane in front of it is partially painted, then in places where the plane is not painted we have a BW character, although its UV is already colored.

The question is next. How to make the plane that lies closer to the camera, unlike the characters, not affect them with its own BW filter?