Issue with using light channels, is this possible?

Hey there,

So I am trying to light my scene dynamically, using primary lights and fill lights to fake GI.
The issue is that my character will get lit by the fill lights as well which causes unwanted effects.

My idea is to use light channels to separate which lights affect which objects, but for some reason this is not really working right.
I get strange artifacts where if I look at an object at specific camera angles it will either receive light or not from unwanted channels.

Can someone explain to me why this is happening, how to overcome it, and or any other solutions to my initial issue of faking GI?