Method to mass-import thousands of materials?

Development Rendering
1

Getting ready to begin bringing in sections of maps into the engine.
These are OBJ files with simple material headers, reminiscent of idTech4 MTRs.

The previous engine used a very concise and straight-forward material script, for example:-


highLevelMaterial "ve10/florfortellicrclrlght1"
{
    template = ShaderBaseMaterial
    alphaRejectFunction = GreaterEqual
    alphaRejectValue = 99
    reflectionColor = 1 1 1
    reflectionSpecificCubemap = "Base\\Types\\Basic\\SkyBox\\env_ifunky1d.jpg"
    emissionColor = 1 1 1
    emissionPower = 2
    specularColor = 1 1 1
    specularPower = 2
    specularShininess = 512
    displacementTechnique = ParallaxMapping
    heightScale = 0.037
    diffuse1Map
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_clr.dds"
    }
    diffuse2Map
    {
        texture = "VE\\Textures\\detail\\detaili_74.dds"
        transform
        {
            scale = 16 32
        }
    }
    diffuse6Map
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_occ.dds"
    }
    reflectionMap
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_rfl.dds"
    }
    emissionMap
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_lmn.dds"
        transform
        {
            scale = 1 0.5
            animation
            {
                scrollSpeed = 0 17
                scrollRound = 1 0.5
            }
        }
    }
    specularMap
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_spc.dds"
    }
    normalMap
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_nrm.dds"
    }
    heightMap
    {
        texture = "VE\\Textures\\ve10\\florfortellicrclrlght1_bmp.png"
    }
}

This is from NeoAxis, derived from Ogre and not too dissimilar.
The list of textures are as follows:-
clr = diffuse
lmn = emissive
nrm = normalmap
occ = ao
spc = specular/metallic
rfl/rgh = reflection-mask & roughness (inverted)
bmp = parallax
There is also a detail-map used in “diffuse2Map” section here, like 99% of all other materials.

The aim is to mass-convert (using Regex/GAWK “translation”) and mass-import these so the map OBJs will be ready to use them.
So far I’ve looked into DAE and FBX, but no success with either of these so far. I figured I could just assign a PBR-like material in Blender (2.79 = “Principled BSDF”), assign it to a quad and export. All this time, I had just assumed it might be that easy, and can definitely handle that on the data-management processing side.

Most of the stuff I’ve pulled from online involve making a material from scratch within the editor, importing each texture and fiddling each option. Because we have almost 7000 of them, this is outright impossible. We absolutely need a solution we can translate and mass-import.

Any advice on a text-based and extensive method to achieve this would be greatly appreciated.

.
.
.

2

The export formats only have the most basic support for materials, especially OBJ. The engine doesn’t have a feature that can assign textures based on the texture naming either. The closest you can do is use the Datasmith exporter which can do conversion of materials.

3

Bad decision. Really hard to do and bloat due to lack of master materials.

4

Thanks, I’ll check out the Datasmith docs and see how close I can get. I realise nothing is going to be perfect, but if we can at least boil it down to functional and just needing fine-tuning, then we’ll be happy enough. Maybe there’s a lot that can be done after the fact, in terms of mass-automation.

When you say “assign textures based on the texture naming”, if you mean a mesh won’t inherit/reference a material based on its header, well, that just seems impossible and unbelievable to me. Every single engine and 3D software I’ve ever used does this. It would mean every mapper needs to manually pull their triangles apart again, in the engine, and manually reassign their textures everywhere, for everything? If so, that is a major flaw of the engine right there.

This is something else I realise, that for optimisation all materials should exist within families of technically similar materials. I do already have them organised something like this, and can moreso, so we’ll just have to accept the initial bloat and work on that later. The most important thing for now is getting our existing content in there, and optimisation will come later.

5

No, it will assign materials if they have they are named properly to what you’re importing, but in your original post it looked like whatever previous engine you used was able to create materials with the textures in the right slots as long as the textures used the right suffix, which isn’t something Unreal can do

6

That’s great to know! I couldn’t actually believe it.

So, no, the matching filenames are all done manually. What I am looking for is a text-based file (can be a model) that will let me import the most compatible and extensive types of materials as possible, at least like that NeoAxis file. Then the goal is to use Regex/GAWK to automatically match that as close as we possibly can to that, and be able to import it into Unreal all at once.

Honestly, the lack of text-based anything is bad, to be forced to clunk around in an editor with nodes and stuff. That’s all great to editing one thing at a time, but, we need to be able to mass-automate things too. I could only hope I’m missing something as simple as some conversion tools. I’m not even sure if I can ALT and TAB around the UI in the editor, so it seems I can’t even automate that. This has never been discussed?

If I go to ‘Asset Actions’ from a material I made, it will export a .COPY file, but I can’t import it again? It says “Failed to Reimport”. This even seems like what I’m talking about but why can’t I import it? What do I need to do to it?

As for Datasmith, I didn’t realise it’s a third-party plugin, and not for Blender. Not actually for any freeware.
Some way to convert .uasset files to and from non-binary formats? I’ve looked into that before, didn’t seem possible.
I strongly feel like there’s an answer to this that could be much better documented.

7

Yes, this is 100% possible to do (I’ve done it before in-game with a custom blueprint obj importer). Since your materials are setup for another engine, you will need to write a custom importer to import them. Luckily, you can do this all in blueprints.

Since your meshes are already obj, you should be able to import those normally with the editor (but you can still use blueprint, too). Your real problem is just the materials and textures.

Engine Version:
The engine version I’m using is 4.20, so things may be different for you.

Plugin:
The only plugin you will need is Rama’s Victory Plugin. It will be used to load in the text files.

Making the Blueprint:
You can either do this with a blutility, or you can make a blueprint class and place it in a level (which is what I am doing).

Create a new blueprint class, and make it’s parent class “PlacedEditorUtilityBase” (this may be different on your version). This is important because it will give the blueprint access to editor utility functions.

If this doesn’t work, you can use the docs to figure out what type of asset to create. Whatever it is, it needs to support editor utility functions.
If you can’t find what to use, you can just install 4.20, import them there, and then migrate the assets to a newer project.

Making Buttons:
To make a button in the details panel, make a function and check “CallInEditor”. This creates a button that lets you run the blueprint in-editor without pressing play. If it has any input parameters, it won’t show up, so to make input variables, just use regular class variables and make them editable.

Importing the Material Files:
Using the Victory plugin, you can use the node “LoadStringArrayFromFile” to load in a text file as an array of strings, one for each line. You can them parse them to get the parameters.

Alternatively, you can convert the material files to csv or json and import them as data tables. This would remove the need to parse them, as well as the need to install the plugin. There are also json plugins available which can import json directly into blueprints, but I haven’t used those.

Importing the Textures:
To import a texture, read its path from the material file. Then, use “ImportAssetsAutomated” to import them directly as assets in your project. You need to supply it with import data, which you create by using the “ConstructObject” node and setting its variables. The only necessary variables to set are “DestinationPath” and “Filenames”. The other variables are optional. The important thing about this node are that it imports files directly as assets rather than just storing them in variables. It also supports every filetype that works with the engine, so you can use this to import other types of assets, too.

The returned value is an array of all successfully imported assets. However, it is **IMPORTANT **to connect it directly to a set (variable) node before doing anything with it. When I connected it to a foreach loop and imported 1 asset, it imported it 3 times. Connecting it to a set node solved this.

Creating the Materials:
Since the materials all have common parameters (ex. diffuse map, normal map, emissionColor, specularPower, etc.) you can setup a base material that has all of those parameters already defined and connected. Then create a material instance of it (right click -> create material instance) and leave it as is. In the blueprint, for each material file, duplicate the base material instance using “DuplicateAsset” and set the parameters using “SetMaterialInstance(X)ParameterValue” (scalar, vector, or texture). This way, you don’t need to worry about setting up a new node graph for each of your materials and deal with the shader compile times.

Alternatively, there are functions where you can actually construct an entire new material graph (create, edit, delete, and connect nodes) solely in blueprint. But this a little more involved and would require shader recompilation for each new material.

Importing the Meshes:
You can either import the meshes using the editor, or you can import them using the same method as the textures. Either will work. I would recommend importing the meshes first, one at a time, importing the materials after each mesh. That way, you don’t have to search for the materials later to assign them as they will be already stored in your intermediate variables. However, if you import materials first, then import meshes, there are nodes you can use to search for the material assets.

Assigning the Materials to the Meshes:
If you decide to import all the materials first and import the meshes using the editor, the editor may be able to automatically find and assign the materials. If not, you can assign the materials through blueprint.

To assign a material to a mesh asset, use the “SetMaterialEditorOnly” node. This node sets the material of the mesh asset itself rather than a mesh in the level.

End:
Save your project and that should be everything.

The attachments contain three examples, none of which use any plugins:
-new material instance: shows how to make new material instance assets through blueprints.
-import files: shows how to import assets through blueprints; these can be any filetype unreal supports.
-import textures and create material: combines the first two methods to show how to import textures, create materials, and assign the textures to the materials.

Blutility:
If you want use blutilities instead, you can possibly simplify the process. Import all the meshes into the editor first. Then make a blutility called “GetMaterialsForMesh” that would do all of the above stuff, but only on the selected mesh.