Bone physics constraints not following parent properly

I’m having issues with bone affected physics that seems to follow their parent with some delay, let me explain with context :

I have a character with big ears, each of them having a bone to make them flap slighly with physics (picture at the end). In my character blueprint I use the “Set All Bodies Below Simulate Physics” node to make the ear bones affect by physics.

I generate some constraints and in their settings, I set the “Linear Limits” and “Angular Limits” to “Locked” which should (from my understanding) make them not move at all, but when I play it works correctly EXEPT the ear struggle to follow the head bone and only align correctly after a few frames (see gif below)

picture of my bones and constraints setup, in case it can help :

Please let me know if you have any clue on what could be the issue !