Oh, I found the solution in another thread:
If you open your Retargeter and select the Root bone then look up in the details panel and search for translation, you need to set the translation mode to Globally Scaled instead of none.