Thanks for taking the time to reply! I also suspect it has something to do with the baking process, but not sure what is happening or how to fix it.
This is what it looked like in substance, so I assumed it was fine. But there is definitely some weird stuff going on in the normal map,
I guess ill try to make a new mesh and see if that fixes it 
Below is the in-engine lit view and buffer visualisation view
