Lightmaps behaving very weird after light build

Hi, I don’t know if I’m missing something, but I pretty much searched everywhere and couldn’t find any solution to this problem.
So, basically, I have this weird lightmap behavior whenever I build lighting. Lightmaps somehow stretch or shift or I don’t even know how to call it.
I’ve attached 2 screenshots (before and after build). I have everything aligned perfectly to the grid and I’m using Blender as you may see. It doesn’t really make much difference in this example, however, I have one much more complicated map where seams around hard edges are pretty obvious.
If you have any clue what’s happening just let me know because it’s driving me crazy.