Importing text files is unnecessarily hard

One would expect that any game engine would let you import text files easily by default, right ? Well, that’s not the case in UE4.

Right now if you want to import something as common as a JSON or a XML you have to create a plugin just for that(well, you first have to search for a plugin that does something similar so you can use it as reference since there are 0 docs about the topic) and then pray so Epic doesn’t decide to use that format for something else.

A few months ago I had to expend two days just to make a plugin to import text files. And now turns out I can’t import JSON files anymore because Epic decided that JSON files should be used for DataTables. Setting ImportPriority in my UTextImporterFactory to MAX_int32 doesn’t seem to solve anything, so it looks like Epic is bending its own rules here.

That’s the kind of unnecessary friction that makes Unreal so unfriendly sometimes.

Ideally, I think importing files should be something like this :

Let us import any binary or text file by default as a standard text/binary asset. Chances are we just want to read some data, but that way we can enjoy from editor integration, auto-imports and auto-reimports.

If there are one or more factories that uses the extension, just ask us what we want to do instead of using the current priority system. Some extensions like JSON and XML are too common and will probably be used for more than one thing on any non-trivial project. In fact, that’s the case with DataTables.

For common cases, let us choose a default behaviour, so we’re not asked every time we import a PNG or a FBX.

I think that would be easier for everyone and doesn’t have any drawback compared to the current behaviour.

Heya,

Thanks for reporting this, we’re going to take a look and try to ensure that the engine level factories that consume JSON either have a low priority and/or only engage when the format of the contents in the file matches what is expected.

If your json is being generated by some other tool or editor that you control, it might also be worth considering using a different extension, although obviously this isn’t always possible.

The disambiguation dialog is a good idea, but it is probably a little further down the road due to the architecture of how the factories currently work.

Cheers,
Michael Noland

I like the idea of Generic Text/Binary Asset. When you drag some file into the editor, editor will ask you to import it as Generic Text/Binary Asset if it’s not already recognized asset format.

Hi Michael, thanks for your attention.

Looks like Paper2D imports JSONs just fine, is Paper2D handled on the engine as a special case ? Didn’t you have to do anything special to make it work on 4.8 ?

The first thing I tried to do was to look at the PaperJsonImporter module since that was what I used as reference in the first place, but looks like you refactored quite a few things in the plugin since 4.7 and PaperJsonImporter is not there anymore.

And yeah, using a different extension is not really an option. I could change it manually every time I save, but yet again, reading a text file shouldn’t be that hard.

The good news is that I found a workaround. If I place a JSON file inside one of the folders that the editor scans for auto-imports and click cancel on the DataTables dialog, then the file gets imported by my own plugin. I’d like to get it fixed properly but at least I can work for now.

@Nate
I could share my plugin if you are interested. The only catch is that you have to add any extension you are interested in manually and that, well, there can be some conflicts with some extensions sometimes, as you can see.
The only reason I didn’t share it when I made it is that since there are no docs on ImporterFactories I’m not really confident enough to support it.

Hey,

I was the one who added the Data Table JSON importing. I’m sorry to hear that it’s caused you this trouble.

I have been looking into this today, but I found that the Paper2D importing was also broken. How are you trying to import your JSON file? I was just drag-dropping the file onto the Content Browser (which should be the same as using the Import option).

With regards to the issue of the factory priority not being respected, I have found one path through the code where the matching factories for a given file extension weren’t being sorted before being tested. This basically resulted in them being processed in the order they were found, rather than based on their priority.

I have a pending fix for the sorting issue, and have also lowered the priority of the Data Table import factory so that it’s below normal by default. Hopefully I’ll get these changes submitted tomorrow, and I’ll follow up on this thread once the changes are in.

Thanks,
Jamie.

Hi Jamie. Great news! Thank you.

Looks like Paper2D only imports JSONs if they are recognized as Tiled maps, if not, I suppose the file is ignored (which I didn’t know was possible) and the DataTable import menu appears. Maybe that’s why Paper2D importing doesn’t work for you ?

I’m using JSONs for Tiled maps too (but I generate 3D tilemaps instead of 2D ones), that’s why Paper2D was working fine for me.

Heya,

As a heads up, if you wanted to use a UPaperTileMap asset and visualize it in a different way that should be doable without creating a separate importer, though it all depends on exactly what you are doing.

Cheers,
Michael Noland

The changes I mentioned before have gone in as b97258a.

This will make the priority work correctly for your factory, but it is also essential that your factory override [FONT=Courier New]FactoryCanImport to perform a meaningful test against the JSON data to make sure it’s something it can import (see [FONT=Courier New]UPaperSpriteSheetImportFactory::FactoryCanImport and [FONT=Courier New]UPaperTiledImporterFactory::FactoryCanImport for examples of this).

Anything not imported by a higher priority factory will simply be consumed as if it were Data Table JSON.

Great! Thank you.

I don’t have access to a proper computer at the moment, but I’ll try the fix as soon as I can.

Will this be included in the next hotfix release ?

No, it’ll be part of 4.9.

Our hotfix releases are only for fixing critical blocker issues (such as unavoidable crashes), and aren’t supposed to make API changes - the change I made to get this working meets neither of those criteria :frowning:

I want to add my two cents. I need to import a text file with some custom data and process it. I can do it using blueprints as it’s a simple document, but I have a LOT of these files. Just being able to add a text file to the content browser would be nice.

It’s 2017, UE should definitely import text files.

Why not use data tables instead?

That is the plan. The plan falls down when you realise UE has no way to import the text file you want to parse into the data table. This sounds like a job for blutilities or something, but the basic text file support should be in there. I’m going to take five minutes to add a factory for it later on today, but it’d be nice if it already existed.

Then why not create the data table outside of UE4 (just save as a .csv file) and import it that way?

Because the data isn’t in csv format. The TEXT needs pre-processing, which if done in a blueprint would be an excellent time to populate it into a data table. Trust me when I say I’ve thought about this beyond knocking two braincells together.

Yes, but UE4 is not a text pre-processing tool. You probably have to write some program to read the .txt files, combine the text files, extract the data you want, add deliminators and output a final .csv file for import into UE4. I have done this before in python, it wasn’t too hard. I think stackoverflow.com is a better place to solve your problem.

So you’re saying I should write a python program to convert it into two different formats just to avoid writing a plugin that converts it directly into the format I want?

Honestly, you’re smothering a reasonable feature request with litigation. It’s not welcome.

I’m saying that you should write a python program (more like a 30 line script) in order to combine and convert your .txt files into .csv files if you want to use the data table features in UE4. Even if you were to import a .txt file somehow you couldn’t turn that into a data table anyway, you need a data table asset which is created by importing either a .csv or .json file. The blueprint nodes used with data tables use this data table asset type, not text or string or any other input type.

This is different from the original feature request of this thread which was simply support for importing of plaintext files, which has nothing to do with data tables. I suggested the use of data tables because you had text files with some information, which usually means you want to use the information contained in them in some logic/character/item/dialog line setup or whatever. You didn’t say what information these files had or what you were intending to use them for so I’m guessing.

I said this because you bumped a two year old thread asking for .txt support, which still hasn’t happened, so I doubt it’s a priority. Instead you could use a data table “workaround” (read: intended workflow), but you would first need to convert your files if they were separate and not proper syntax. Normally you would create these files in Excel or something and just export as .csv and UE4 would import them fine.

You said my posts weren’t welcome, and my response is: who are you to say that? You didn’t create this thread, and it looks like you’re requesting another feature entirely. I posted this so you and others could find a way to solve your problems by yourselves, which is what I thought you wanted to do. If you don’t, you are free to wait for another two years. Do as you wish.

I feel like this thread is no longer about importing text for use in the editor and more about someone desperate to not be wrong. Let’s leave it here and try to learn from this.

I agree:
text.gif