Flip Green Channel or not?

Is there a sure way to determine whether or not do I have to flip normal’s green channel?
I know it sounds silly, but sometimes it’s really difficult to tell.
For example, I have a stone surface texture I downloaded from somewhere a while ago, and just by looking at it I can’t tell if flipping the channel makes it correct or not.