I want 2 vectors representing the bounds of a mesh but when I rotate the actor, the bounds are wrong when the object rotates

I’m currently trying to make a board, where I can drag objects along a flat plane. I also want this actor modular, so I’m allowing user’s to replace the existing mesh and use another.

I get the bounds of the component, to get the min and the max vectors to make a box. When an objects is dragged, it’s location will be limited between those two vectors.

My problem is, whenever I rotate the actor, the the component bound vectors that are found aren’t the corners of the mesh.

The diamonds are the points found.
Here’s at yaw 0

Here’s at Yaw 90

I’ve tried using rotate around axis, but nothing I do seems to fix this problem