Hello guys, I fixed the problem 
So to start of I got my inspiration from macxike on this topic Relative rotation with physics constraints
but made something a little bit different.
First problem is our doors rotation value is wrong. So to get the correct rotation we create another dummy door that is invisible. And on event tick we get the world location of the original door, set dummies world location on that. Now they are moving together.
Next we use dummies relative rotation to do what we want to do. Here next problem comes. We cant set relative rotation of our door because it is simply incorrect. So we add actor rotation to our relative rotation to make it world rotation and set it there.
My BP has some more stuff that uses relative rotation and we use dummies relative rotation on that.
So it is working pretty nicely. Idk if this is good for performance or what but it gets the job done.