UTF Encoding driving me to insanity

OK I’ve been tearing my hair out for a few days on this, the long story short, I’m trying to take the new SQLiteSupport Module, and use it in IOS. From what I have gathered, there is no reason why SQLITE should not be able to work with iOS. Now with the issues…

The basic issue, from what I have gathered is, TCHAR * does not play nicely with SQLITE’s interuptation of UTF-16LE. Basically, if I create a FString, with X file name, then pass it to sqlite3_open16 on IOS, it will either fail completely, or create a file with a bunch of junk encoding at the end, and it won’t even be a UTF-16, instead pragma will return UTF-8.

If I try to open an existing database in UTF-16LE, ill have issues opening it, and if I’m lucky sometimes it does but the data coming back from SQLITE3’s UTF-16 is just questions marks or other encoded junk.

What am I doing wrong? Is there something about UE4’s or Sqlite’s idea of UTF_16 I’m failing to account for? Do I need to remove or add bytes somewhere?

Someone will be my hero if they can give me any pointers here. My final idea is just to go UTF-8 and rewrite the SQLiteModule to be UTF-8 instead instead of 16 as currently implemented, as earlier it seemed I had better success converting UE4’s strings to UTF8.

Never tried it myself, and I don’t know about the SQLite module you are using, so just from the point of converting strings to utf-16, it looks like UE4 itself has no option to do that, I at least couldn’t find one, which means you might need to use the string class that comes with the operating system and hope that it can. Take a look at FPlatformString, I don’t know what it references on iOS, on mac it uses FApplePlatformString, which can convert to a CFString (Core Foundation) on iOS it might be an NSString, though i don’t know.
You theoretically can tell a CFString to take another encoding than by default.
https://developer.apple.com/library/mac/documentation/CoreFoundation/Conceptual/CFStrings/Articles/UnicodeBasis.html
https://developer.apple.com/library/mac/documentation/CoreFoundation/Conceptual/CFStrings/Articles/Converting.html

maybe it helps, but probably people who actually use the sqlite module themselves have better answers

linky: (ill be updating on the answer page for now on)