Datasmith FBX is only there to import FBX from VRED and Deltagen that are Automotive softwares. Datasmith FBX will analyse the FBX header and if it is a “normal” FBX will not import it.
Blender FBX should be imported as a normal FBX in UE. If you need to import as a scene then go for File>Import into Level.
There is no Unreal exporter for Blender to udatasmith. there are some addons to export from blender to UE but not sure how good they are and if they are not too much for what you are trying to do Download our new Blender addons - Unreal Engine
Regarding glTF import, from 5.1 onwards you should not use Datasmith anymore. Just import directly: drag/drop in content folder, right-click import in content folder, or file>Import into level
I would try import into level of FBX, else import as glTF or USD. Those are good alternatives to FBX.