Hollow Sphere Normal Baking

I’m having an issue baking a good normal map from a hollow sphere I made. (Cage/LP/HP)

Here is the UV layout (which I very well could have done wrong). I created seams on the inside of the holes and use spherical projection mapping to get the two UV islands from there.

And here is the resulting normal from xNormal (Had to post a link because I apparently can’t add a 3rd photo to this post)

What can I do to fix this?

Edit: The spherical mapping is causing some pretty gnarly distortion, so I’m gonna have a go at fixing that.

Do you really need to have it as geometry? Masked mesh won’t do ?

Yeah I’d prefer it to be actual geometry. I’m assuming that the UVs are causing this issue, but admittedly UVs are my weakest area and I’m not sure how to fix them.