Okay, if someone else stumbles upon the same problem: I was trying to get the bounds from my static mesh component directly… this gives weird results when rotation the actor that owns the static mesh component.
Solution: Fron the static mesh component get the static mesh and get the bounds from the static mesh (not the component). I then multiply it with the relative scale 3d of the static mesh component (you can skip this if your mesh is unscaled).