An idea that might be worth exploring could be in a post process material calculate luminance and use it as a mask:
Hope it helps.
