Yeah, I just spent 5 days trying to narrow down this issue, and I have some workarounds, but no answers as to why this happens.
Firstly, I’m assuming that you’re exporting your FBX’s with the version set to: “FBX 7.4 Binary” correct? That’s the source of the scale discrepancy causing the animation to be a different size than the skeletal mesh. There is no way around it when using the 7.4 Binary (at least, not that I’ve discovered in the 5 days of constant tweaking then exporting and importing… over and over again)
I’ve been using that for my project because of the animation bake settings that it has, but it (for what ever reason) rescales ONLY the animation files and not the skeletal mesh or the skeleton (wtf? am I right?). What makes this more confusing is that everything is perfectly fine when you re-import them back into blender.
To avoid this headache, use the “FBX 6.1 ASCII” version setting. Set “Y forward” and “Z up”
It has it’s own quirks that need to be handled in UE4 on import (like disabling “Convert Scene” in the import dialog box), but at least it doesn’t cause this odd rescaled animation problem we get using other FBX versions.