Pain of Unreal Engine: binary assets

I’ve been making maps for months now and then I read this stuff when I have a problem when a umap wont load, as it thinks it a uasset.

If this is indicative of the above problems, then I guess the next answer is, are we going to get honest answers as to have to prevent corruption , IF that’s what I’m looking at here ?

I have many backups, so if I have to I can go one down and work back up , but I’d rather know the status because its been months that I"ve been editing, and I don’t recall having any problems until moving to 4.14.

Is this more likely to happen in the newest engine release , if so I should ignore it for now ?

ty

Text-based assets would make them MUCH easier to debug, and the ability to diff in source control would be an added bonus. However the key reason is preventing corruption - I’ve seen the “engine thinks umap is uasset” once and I sincerely hope, never again!

I suggest a text based asset design, similar to what we use in DirectScene. It’s pure Json, optimized to be easy readable and editable.
Subnodes of the object can be defined inline as usual, or as an external reference, in case they are too bloated or if they are binary files.

Example Scene
As you can see in the following example, assets are divided in N files (only 4 of N files are listed here, material, tracks and regions are left out for clarity)
For example, the mesh reference just links to a binary object, this is the original unreal asset.

With this system you have an extremely clear view of what happens in your assets and you can work inside any editor to text copy parts, using find, replace etc.

Scene.Waldhaus1


{
	"Scene": {

       "Description": "Testscene for Grand Hotel Waldhaus",
        "FloatTest" : 3.1415,
        
		"Version": "1.0.0",
		"PlayerStart" : [0.0,0.0,200.0],
		
        
        "Tracks": 
                   "Track.Floor1",
                   "Track.Cube1",
                   "Track.Camera1",
                   "Track.Boy6"
                   ]
    }

}

Track.Boy6


{
   "Track" : {
      "Name" : "Boy6",
      "Actor" : "Boy1",
      
       "Loop" : 1,
       "Speed" : 1,
       
       "Regions" :
       
        "AnimationRegion.Boy6"
        ]

   }
}

Actor.Boy1


{
	"Actor": {
		
			"Name" : "Boy1",
			"Type" : "Skeletal",
			"Mesh" : "Boy",
			"Material" : "",
			"Location": [100.0, 0.0, 100.0],
			"Rotation": [0.0, 90.0, 0.0]

		}
}

Mesh.Boy


{
   "Mesh" : {
      "Name" : "Boy",
      "Path" : "/Engine/shared/Meshes/boy/boy_SKELMESH"
   }
}

Just saw that it’s finally on the roadmap for 0.15: Trello

But no details as usual…

Please just change binary uasset to text uasset for developer git compare, text uasset download from marketplace

Bumping to see if there is interest for a new asset format which is flexible like glTF where the actual asset file could be text or binary or kind of mixed.
Contained data can be inline or just a reference to an external file.
During development, I would love to use original files, for example simple wavefront meshes or plain image files, shader as text etc.

https://forums.unrealengine.com/development-discussion/content-creation/1415582-the-new-gltf-import-supported-in-4-19-preview-is-awesome-experimentation

Definitely would like to see text-based asset files where possible, too. Many excellent points have already been given here why this would improve UE and the workflow with it. I agree that a hybrid approach like with glTF would be good as I can see why sometimes binary may be preferrable by the engine.

Epic is definitely working on this: just search for any commits that contain “TBA” (Text Based Assets). When it’s going to be done is another story.

As a software engineer of 40 years, please do not go with JSON. It is designed SOLEY for machine-machine communication. Instead, if a human is EVER going to react to something (and with diffs, they do) PLEASE use YAML.

At my last job JSON was used EXTENSIVELY and since you can’t comment in it, we ended up with variables with similar names but wholly different meanings. Long term, YAML is good for humans AND machines whereas JSON is only good for machines, especially for large projects.

Just use a JSON parser that supports comments… most of the populer C++ ones do (jsoncpp, rapdjson, even nlohmann::json gave in not too long ago). Just because those in control of the JSON spec are too narrow minded to see the value in comments in a json file doesn’t mean you can’t use it. Now if you’re going to send that json data to some external service, probably a good idea to strip comments out first.

Or whatever other text format, I agree any is better than binary.

On the more broad topic of binary assets in Unreal, pretty pretty please on the TBA / text based assets - they would solve so many of our team’s issues when it comes time to merge. A very simple change in blueprint, eg a default value changing, can easily be merged via text whereas impossible when saved as binary. It’s so much that we don’t use branching because the binary assets make it too much of a PITA to get a dev branch merged back into main. Not good at all.

1 Like

I think you misunderstood — I’m quite aware of the bastardized JSON parsers.

Comments are not part of the JSON standard définition. If you are using a JSON parser that allows comments, you are using a non-standard parser.

YAML allows comments by design. It is more about understanding the tools and knowing which to employ.

My point is if Epic wants to CORRECTLY fix a lot of their system, they really should move to YAML.

[=“anonymous_user_15c8a15e, post:32, topic:24394”]
A very simple change in blueprint, eg a default value changing, can easily be merged via text whereas impossible when saved as binary.
[/]

Agree wholeheartedly. Binary assets like these are a pain.

1 Like

I was copying blueprints and whatnot on the Mac and discovered that they ALREADY are in some form of ASCII…

Begin Map
Begin Level
Begin Actor Class=/Game/ThirdPersonBP/Blueprints/ThirdPersonCharacter.ThirdPersonCharacter_C Name=ThirdPersonCharacter_167 Archetype=/Game/ThirdPersonBP/Blueprints/ThirdPersonCharacter.ThirdPersonCharacter_C’/Game/ThirdPersonBP/Blueprints/ThirdPersonCharacter.Default__ThirdPersonCharacter_C’
Begin Object Class=/Script/Engine.CapsuleComponent Name=“CollisionCylinder” Archetype=CapsuleComponent’/Game/ThirdPersonBP/Blueprints/ThirdPersonCharacter.Default__ThirdPersonCharacter_C:CollisionCylinder’
End Object

etc etc etc etc etc

I accidentally did a paste using UE5 so I went and double checked with UE4…

Bottom line is that for cut and paste, Epic already has code that does this! It just hasn’t been turned into files. Why? I’m guessing size.

I didn’t spend much time on it because other things… but I thought folks might find it interesting that Epic already has the functionality in the code, they just don’t store files this way.

EDIT: Also note this appears to be a VERY old style (probably before XML) of serializing objects. Not a criticism as it looks easy peasy to parse. Just surprised that no one has thought “Huh, I wonder if people would be interested in saving projects in ASCII rather than binary?” Then, for the build, turn it into the binary.