Now, std::string(TCHAR_TO_UTF8(*FileOnDisk)) seems to be the standard way to convert an FString to std:string in UE4 (it comes from a Rama tutorial and is the standard answer found on google), but it doesn’t seem to work well with windows file name FStrings.
If i enable the debug message here, i get a wrong file name also: “hé hé” becomes “h?? h??” (and using TCHAR_TO_ANSI gives “h? h?”, so each “?” is one wrongly encoded byte apparently).
Any idea of how to handle that ?
As always, i ask in case anyone knows, and will post the answer if i find it.
The whole idea of IFileManager is to have a way to handle files platform independent. It won’t return ever a character set different. Which platforms you intend to run the code? PC with Windows only? If just Windows you can ignore IFileManager and use Windows SDK API to handle the find files for you, which will take care of the code page (character set) in use. If you want to use the code in any other platform support by Unreal you might just don’t allow anything in a different character set for the file names.
I never any issues with unicode file names. Just note that the code in that repository is the partial implemention since I have refactored things a bit and evantualy will upload. But, if you take a look at my methods they accept FString as parameters and as return values, while the underlying code implemented using Boost uses std::string. I did the conversion using StringCast.
@NelsonLima: thanks for the IFileManager info. Yes, my game is for PC/windows only, so if i don’t find a better way, i might end up using the Windows way to list files, but i’d like to do that with standard UE4 tools, like they’re supposed to^^.
@NuLL3rr0r: StringCast seems to protect me from the risk of a null ptr, but if (big if there^^) i understand correctly, that wasn’t my problem using the macros since i was getting a file name “close enough” to the original one. And indeed, when using StringCast like so:
auto Conv = StringCast<ANSICHAR>(*FileOnDisk);
const std::string FileOnDiskStdStr = Conv.Get();
I get exactly the same result (crash with a wrongly encoded file name, all special chars encoded on 1 byte as expected from ANSI: “hé” becomes “h?”).
And StringCast won’t compile with WIDECHAR, UTF8CHAR not UTF16CHAR.
But i believe std::string is waiting an ANSICHAR anyway, so i guess ANSICHAR should be the right one, although here it’s not working.
So although i see the benefit of using StringCast over macros and thank you for this info, the use of StringCast doesn’t seem to solve my problem unless i’m using it incorrectly.
I take a lot of language precautions because i’m in complete personnal uncharted territory here so i might misunderstand things
Going on with my researchs, still opened to suggestions, and of course will post any progress
Hi, my post is not a solution but a suggestion to rethink the cause of the problem. Instead of storing the special characters in the file name, why not either
a) write without special chars and use a replace convention, e.g. "whenever I have <é> I will replace with e_, <ù> becomes _u, so that you can save the file then you switch back to your chars on read, or
b) save as “Shaman he he.csv” and attach some metadata to a separate Shaman he he.meta file where you store character variables, e.g. characterName=“Shaman hé hé”
Thanks for your clever and original suggestions but they appear a bit impractical in my case and raise other problems:
i should have to keep my own map between official encoding (é) and my own encoding (e_), and given the number of special characters around, maintenance would be impossible: it amounts to create my own encoding system^^.
the files are named after the names of the character they contain and are searched according to this name
then, i need my files to be user readable, as i export them in csv to allow users to edit them in a 3rd party software
I wouldn’t want to impose to users a rule looking like “if your character is named hé, you’ll find it in a file named he_.csv”
I’m not even sure that what the debugger displays is what the AddFile function is seeing, because the debugger has its own way of displaying characters (i get something different with AddOnScreenDebugMessage: “hé” becomes “h?” with TCHAR_TO_ANSI and becomes “h??” with TCHAR_TO_UTF8 and both are different from the characters displayed in the debugger.