Saving assets as text

General Feedback & Requests
1

Hey guys,

Please serialize assets as text by default, or at least give us the option, so that they can be diff’d and merged. Scenes, blueprints and others are currently being saved as binary, which would cause issues when working with a team if you use branching, but also makes it impossible to quickly check for yourself the changes you’ve made in between revisions.

Thanks!

2

Did you know that there is a diff tool built into the editor?

Of course that doesn’t help when you’re diffing with external tools, but it’s still a nice feature.

3

Thanks Bajee. I know that exists, but does not help when using something like git, nor when reviewing what happened on a changeset without checking out and going into the environment. It also does not help if you need to merge assets across branches. It’s a stopgap measure, not a solution.

2 Likes
4

Well, a textual representation would certainly help, I agree. Maybe as an option, levels, BPs etc. could be saved as XML or something like that?

5

Or something more human-readable. :slight_smile: Unity does YAML, but JSON would work too. Things should be serialized in a consistent order for it to be useful: Unity does this idiotic thing where apparently scene item serialization order is completely arbitrary, so if you save the same scene twice you end up with significantly different resulting files.

1 Like
6

Hmm, I’m not sure EVERY asset should be serialized to some text format, but certainly anything involving setup/configuration of objects, behaviors, worlds etc. So blueprints for example I think should have an external format, plus the world setup, object component setup etc.

I’ve asked for this same thing and the last twitch stream suggested that it IS possible in that they demo’d a pastebin style site which shared blueprint code. It seems completely do-able considering there is a reflection mechanism in place. My biggest concern isn’t necessarily about diffing assets (although that would be useful), its more about the potential for recovery from disaster. For example I recently had a screwed up scene in Unity that I tried to recover from source control, only to find that the mechanism Unity uses to serialize scene files doesn’t like missing references, basically a ton of my objects became invalid in the scene because of some internal ID changes and that destroyed about a weeks worth of work.

My point is, that say for a C++ file, I know that once that is in source control, if I ever need the project to recover from some kind of catastrophe as long as I have a recent backup of the source control version I can recover to that last save point. With a binary serialized version of assets, it might be that assets store references to other objects and hence if anything changes with regard to object ID’s or versioning I’m on very shaky ground.

I guess I might be paranoid after the Unity issue, maybe blueprint doesn’t share the same potential flaws. But given a few people on the forums have already said that their blueprints were corrupted and lost work due to that, it does concern me a great deal.

Maybe a post going over how to correctly ensure a build remains properly stored in source control would be useful Epic? I’d like to know what does and doesn’t require source control saving and what is automated for example. Does that exist?

7

I’m sorry to revive an old thread, but I can’t believe this hasn’t happen yet.
I think everything should be able to be saved as text. Binaries are hell to use with a VCS, especially with something distributed like git.
The Unreal devs wouldn’t have to do much, just save e.g. binary blueprints in a cache folder and save the text representation in the normal directory. Then the text file has to be updated when the blueprint is modified and saved, and the binary file is recompiled when the date of change of the text file is newer than the one of the binary.
I would do it myself, but I can’t find the serialization logic in the github repo

1 Like
8

This was requested before content as plane text files that allow to edit and program over it too

But don’t see any time soon

9

I think there are two things that make the text-asset approach a bit unfeasable.

First, I dont think that a text format is necessarily “human friendly”. (Just “read” a sufficiently complex ascii fbx file).
And fbx is a rather “simple” structure.
My guess is that any attempt to edit such a UE4 asset text file would very likely corrupt things.

Secondly, unlike formats like fbx, UASSETS have no real format. They just mean serialized data.
What gets serialized and in what order can freely change between versions. There is no formal format specification.
So the exact same material node setup could produce two entirely different text files, depending on the engine built.

But for end user convienience, if implemented, I would prefer XML over JSON.
XML may be a bit more verbose, but we are not living in the ages of 360k floppies anymore.

10

I prefer JSON over XML on this one. It is not space issue, but JSON is a lot more human readable and XML is sometimes too convoluted.

11

This is still on the roadmap for the coming months:

Though it seems to be taking way longer than expected, it was already on the roadmap for december last time…

1 Like
12

6 years later and we still do not have the option :see_no_evil: :see_no_evil:

1 Like
13

I did some research into this and implemented two related features for my project:

  1. When saving a .uasset file it will dump it to a text file (using the same API that the editor’s diff tool uses) that sits right next to the .uasset file. This way you can commit it to version control and your history will show somewhat useful information. Keep in mind this doesn’t work well for some Blueprints as I’ve found occasionally a bunch of stuff will change in the text representation even though I made one tiny change to the Blueprint.
  2. When right clicking on an asset in the context drawer there’s a “Dump to text file” menu which will dump it to a temporary file and open it in your default .txt editor.

Here’s the code in a UE plugin. Note that right now #1 above is filtered to only run on UMassEntityConfigAsset but you can easily change this in the TODO below.

#include "ProjectM.h"

#include <ContentBrowserModule.h>
#include <AssetToolsModule.h>
#include <Kismet/KismetSystemLibrary.h>
#include "UObject/ObjectSaveContext.h"
#include <MassEntityConfigAsset.h>

#define LOCTEXT_NAMESPACE "FProjectMModule"

static void DumpToTextFile(const FAssetData& SelectedAsset)
{
	UObject* Asset = SelectedAsset.GetAsset();
	FAssetToolsModule& AssetToolsModule = FModuleManager::Get().LoadModuleChecked<FAssetToolsModule>("AssetTools");
	FString TempFilename = AssetToolsModule.Get().DumpAssetToTempFile(Asset);
	FPlatformProcess::LaunchFileInDefaultExternalApplication(*TempFilename);
}

static void AddDumpToFileMenuEntry(FMenuBuilder& MenuBuilder, const TArray<FAssetData> SelectedAssets)
{
	MenuBuilder.BeginSection("ProjetM Asset Context", LOCTEXT("ASSET_CONTEXT", "ProjectM"));
	{
		MenuBuilder.AddMenuEntry(
			LOCTEXT("DumpToTextFileLabel", "Dump to text file"),
			LOCTEXT("DumpToTextFileTooltip", "Dump asset to text file"),
			FSlateIcon(FAppStyle::GetAppStyleSetName(), "Icons.Edit"),
			FUIAction(FExecuteAction::CreateLambda([SelectedAssets]()
			{
				DumpToTextFile(SelectedAssets[0]);
			})),
			NAME_None,
			EUserInterfaceActionType::Button);
	}
	MenuBuilder.EndSection();
}

static TSharedRef<FExtender> OnExtendAssetSelectionMenu(const TArray<FAssetData>& SelectedAssets)
{
	TSharedRef<FExtender> Extender = MakeShared<FExtender>();
	Extender->AddMenuExtension(
		"CommonAssetActions",
		EExtensionHook::After,
		nullptr,
		FMenuExtensionDelegate::CreateStatic(&AddDumpToFileMenuEntry, SelectedAssets)
	);
	return Extender;
}

void FProjectMModule::StartupModule()
{
	FContentBrowserModule& ContentBrowserModule = FModuleManager::LoadModuleChecked<FContentBrowserModule>(TEXT("ContentBrowser"));
	TArray<FContentBrowserMenuExtender_SelectedAssets>& CBMenuAssetExtenderDelegates = ContentBrowserModule.GetAllAssetViewContextMenuExtenders();
	CBMenuAssetExtenderDelegates.Add(FContentBrowserMenuExtender_SelectedAssets::CreateStatic(&OnExtendAssetSelectionMenu));
	
	UPackage::PackageSavedWithContextEvent.AddStatic(&FProjectMModule::DumpAssetsToTextFileOnSave);
}

// We do this so that we can better leverage git on various .uasset binary files.
/*static*/ void FProjectMModule::DumpAssetsToTextFileOnSave(const FString& PackageFilename, UPackage* Package, FObjectPostSaveContext ObjectSaveContext)
{
	TArray<UObject*> Objects;
	GetObjectsWithPackage(Package, Objects, false);
	FAssetToolsModule& AssetToolsModule = FModuleManager::Get().LoadModuleChecked<FAssetToolsModule>("AssetTools");
	IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile();
	for (UObject* Asset : Objects)
	{
		// For now only do this for MassEntityConfigAsset. TODO: Add more types here.
		if (!Asset->IsA<UMassEntityConfigAsset>())
		{
			continue;
		}

		FString TempFilename = AssetToolsModule.Get().DumpAssetToTempFile(Asset);
		FString AssetPath = UKismetSystemLibrary::GetSystemPath(Asset);
		FString TextFilePath = FPaths::ChangeExtension(AssetPath, TEXT("txt"));
		const bool bDidCopy = PlatformFile.CopyFile(*TextFilePath, *TempFilename);
		check(bDidCopy);
	}
}

void FProjectMModule::ShutdownModule()
{
	// This function may be called during shutdown to clean up your module.  For modules that support dynamic reloading,
	// we call this function before unloading the module.
}

#undef LOCTEXT_NAMESPACE

IMPLEMENT_MODULE(FProjectMModule, ProjectM)