I suspect a collision related issue between the door and the frame. When physics objects are intersecting on begin play, a large depenetration impulse is applied to them, and you may see some taking strange positions or even flying away. After a while the constraint is able to do its job of bringing the door back in place and the door goes back toward the frame but this time from the outside, so it won’t intersect.
You can temporarily remove the constraint and position the door properly within the frame. Then use Simulate to make sure it is not flying away.Once you are happy with the door position, press K during Simulation to save its location to the Details panel. Now re-add the constraint, it should work as expected.