Blender Normal baking from floating geometry

Hi, I am currently trying to bake some normal maps but Ive encountered some resistance in my way of wanting to do it.
I know how to bake onto a single plane, or use a HighPoly to a LowPoly mesh as long as they are , for arguments sake a HP cube with damaged corners baked to a LP cube just “beneath” it.
My problem is I want to bake the normal from floating geometry onto a single face/polygon, but is seems they are “reflected” or whatever the correct term might be onto other faces/polygons.

The reason for this is I want to be able to create HP stones that gets normal baked onto the sides of a specific building.

Example of my problem

Any help in resolving this issue, a workaround or just guiding me to a better way to achieve this would be greatly appreciated.

Kind Regards,
J

Maybe I just answered my own question with just baking the floating geometry onto planes and later assembling then into a single object x)

Not an expert on baking in blender but I think you just need to make sure there is something for the baking to find on the other side. Looks like you are baking from the floaters only so on the back side the closest “target” objects are actually the floaters on the front.

To fix it just to put a “highpoly” box around your lowpoly box and bake from it and the floaters.

There is probably some max trace distance you can play with so that it won’t find the floaters on the other side but in more complicated cases that probably will not work as well.

Yeah your tip made me realise that I hadn’t even configured the Distance and/or Bias options for the bake in blender, silly me.
For anyone else reading this, either split the model into several parts or adjust the baking settings so that other sides do not bake the same geometry.