Dont you also have all sorts of light issues since you are using a skeletal mesh?
Your issue could be the animation itself.
Re the “how without physics” - easily and much more reliably.
A preset animation will alwas work right, look the same and be consitent.
Once you add 3 or 4 different ones and randomize which you fire off, the user can barely tell if its true or not - ofc, other objects will bounce back from it instead of swinging it open…