You are dealing with bad starting poses.
To get retargeting to work correctly you’d have to copy over the initial rotation of the mannequin’s skeleton onto the animations base skeleton or viceversa.
There’s a process in blender for doing this illustrated here by moving the mannequin to a TPose for creating, and back to the regular A pose for export/usage.
Honestly, all your other options are going to be bad.
You could set your skeleton retargeting settings to animation scaled for the bones, or to skeleton depending on which ones work best, but it wont really work right…