How do I correctly use a normal map with sprites?

Development Rendering
1

Hello.

I’m trying to use a normal map with my sprite to give it some extra shading, and make it receive light better.
This is what I got at the moment: Screenshot - dec9312455b7141e8bca3210c493f27e - Gyazo
But when I turn the character to the left, the shading changes. Like this: Screenshot - 5cb99b182335afba1971320309008640 - Gyazo

I guess this is due to the normal map changing direction.
Any ideas on how to fix this?

EDIT: I have a point light at the staff and at the hand.