Hmm, seems like I found the cause atleast. It only happens when the item in question has a scale of uneven number. For some reason it centers on the middle so if the wall is 7cm wide it centers it and therefore is 3,5cm to both sides = ½cm off-grid. At 8 though it’s 4 cm on both sides = on-grid.
So there comes the next question. Can you turn off auto centering somehow?