Physics Constraint not rotating properly?

This is how my cabinet looks before any rotation:

And when I try to rotate the cabinet, the physics constraint box doesn’t rotate correctly:

Any idea why?