There’s exactly zero difference once imported, as far as UE4 is concerned. It doesn’t matter whether you import an FBX ASCII file or an FBX Binary file into Unreal Engine. Unreal Engine stores FBX files in its own format on import, so they take up the same space on disk.
Longer, slightly more technical explanation of the answers:
Once Unreal Engine imports an FBX file, it stores it as a .UASSET, which is a file format specific to Unreal Engine. Technically, it generates anywhere between one and four .UASSET files from a single FBX. How many files are generated depends on whether you choose to import a mesh, an animation, a skeleton, and/or generate a physics asset. The result is the same whether the FBX file started as a binary or an ASCII file. However, an ASCII file may take slightly longer to import simply because it’s larger. A .UASSET file may also end up larger than the original (binary) FBX file because it stores additional metadata that is not present in the original FBX file.
You can verify this by exporting a single character twice; once as an ASCII, and then again as a binary with no other parameters changed. Import these into a project using the same settings for each. When you examine the .UASSET created within the content folder of that project, you’ll notice that both of them have the same size on disk. If you export an FBX from a .UASSET file, it will export in binary format, even if that file was originally an ASCII FBX. That’s because it is not simply regurgitating the original FBX file’s data. The engine creates a new FBX file based off the .UASSET data.
If you do notice any file differences, it’s because Unreal Engine stores the file path of the original file. So if you import a file named “Binary.fbx”, it may technically take up less space than a file in the same directory named “ASCII.fbx”. This is because the word “Binary” is larger than the word “ASCII”, everything else being equal. It has nothing to do with the format. You should not optimize for this, however. Someone who is that concerned about space optimization can just remove the source file reference entirely.
Nitpicking of the technical explanation:
Technically .UASSET isn’t a “file format” at all. File formats are more standardized, but .UASSETs are not. Its just the extension given to the serialized form of various independent data types stored by UE4. You could say that UE4 converts FBX files to a binary format during the import process; but it isn’t stored the same as a binary FBX. Rather, it’s stored as a serialized .UASSET. I only used the term to make it clear that a .UASSET file cannot be treated as if it were an FBX, because it isn’t one. You can’t just open up a .UASSET file in 3ds Max or Maya. So, referring to it as a separate file format will suffice for the purposes of this post. If you’re curious about the specifics of how UE4 stores information, you’re welcome to examine the ULinker code for yourself.
Which one is better?
That depends on what you’re doing with the file prior to importing it into UE4. Binary FBX files can only be read by programs. However, ASCII is human-readable. You can open it up in Notepad++ and edit it like a text document. You can copy and paste it. You can upload it to PasteBin to share. You can search for and replace strings, such as bone names. If you don’t need human-readability, it’s better to use binary because it takes up less space and is likely to import slightly faster. If you’re storing the original files (and you should!), then that space can add up quickly! It is trivial to convert back and forth between the two, if your needs ever change.