Disable Ambient Cubemap for character?

Any idea how to make the Ambient Cubemap in UE4’s Post Process Volume IGNORE the character?
Basically if it’s turned on my character is super duper shiny, but if I turn it off the world looks rubbish.

My character texture is completely rough with no shine to it so this should not be happening. Help! (it won’t let me upload images sorry)