JSON Conversion of TMaps of UStructs Can't Deserialize

anonymous_user_33e74471 · March 16, 2016, 9:10pm

This is half a bug/feature request. I’m working on 4.8, but I can see that the problem still exists in later versions of the engine. The real problem is that TMaps aren’t really well supported in json falling back to the default export as string functionality. TMap support for the JsonObjectConverter would probably also fix this bug.

Anyway, the bug is that when you try to serialize a TMap of UStructs where one of the Structs in the TMap is the default value of the UStruct it gets serialized like this:

“MyMap”: "((MyKey, ))

With an empty value for the value stored in that key. This happens because the default text export only exports values that are different from the default object, so if you are the same as the default object you don’t serialize anything.

When you try to deserialize the data, it fails and causes the serialization to fail.

This bug is kind of twofold in that the object gets serialized improperly, but also in that a property deep in a nested ustruct shouldn’t cause all the deserialization to fail, it should stay isolated to whatever scope in the json it happened to be nested in unless the json is poorly formatted.

edit: Adding sample project:

Here’s a link to the project:

Here’s a link to just the source directory:

The only relevant changes from a base C++ project are in the source directory, so if you download from there it’ll go much faster (2kb vs 200mb).

anonymous_user_a08c0bf2 · March 23, 2016, 4:50pm

,

Could you please verify that you’re able to reproduce this issue in 4.11 Preview 8? And if you are able to, could you include the exact reproduction steps or a sample project with your next reply?

anonymous_user_a08c0bf2 · March 29, 2016, 1:23pm

Hey ,

We have not heard back from you in a few days, so we are marking this post as Resolved for tracking purposes. If you are still experiencing the issue you reported, please respond to this message with additional information and we will offer further assistance.

Thank you!

anonymous_user_33e74471 · April 5, 2016, 3:52am

Hello! Sorry for the delay. Didn’t get a to do this till now.

Here’s a link to the project:

Here’s a link to just the source directory:

The only relevant changes from a base C++ project are in the source directory, so if you download from there it’ll go much faster (2kb vs 200mb).

I’ll update this in the OP too.

edit: Just for anyone else who finds this, check comments on the OP above for the answer.

anonymous_user_33e74471 · April 5, 2016, 3:53am

Updated with project.

anonymous_user_33e74471 · April 8, 2016, 2:30pm

Just wanted to bump so this doesn’t get lost.

anonymous_user_a08c0bf2 · April 8, 2016, 9:33pm

,

I pulled down your project and tested it in 4.8.3 as well as 4.11.1. In 4.11.1, I did not receive any errors when packaging for HTML5. In 4.8.3, I was unable to even package or launch your project successfully.

Please provide reproduction steps.
Try this on a new project.
Test your project in 4.11.1.

Thank you!

anonymous_user_33e74471 · April 8, 2016, 9:48pm

It’s not a packaging problem. It’s a Json Serialization problem. If you open it in the editor and play you should see debug text pop up that says something like “Json Serialization Failed!” after the following code that serializes then deserializes a ustruct fails.

FJsonObjectConverter::UStructToJsonObjectString(FTestContainer::StaticStruct(), &testInput, jsonString, 0, 0);

	UE_LOG(LogTemp, Warning, TEXT("Json: %s"), *jsonString);


	FTestContainer testOutput;
	if (!FJsonObjectConverter::JsonObjectStringToUStruct(jsonString, &testOutput, 0, 0))
	{
		UE_LOG(LogTemp, Warning, TEXT("Json Serialization Failed!"));
	}

edit: Sorry I switched the debug text to a UE_LOG so there was no popup But the problem should still be evident in the logs.

anonymous_user_33e74471 · April 13, 2016, 3:21pm

bumping again o.O

anonymous_user_a08c0bf2 · April 15, 2016, 1:56pm

Hey ,

We are currently investigating the issue that you are reporting. Please do not bump the thread unless you have additional information to enter about this specific HTML5 issue.

Thanks!

anonymous_user_33e74471 · April 15, 2016, 3:02pm

It’s not an html5 issue. It’s a bug with the json object converter, but I will stop bumping. Thanks for the update.

anonymous_user_a08c0bf2 · April 22, 2016, 5:45pm

Sorry for the lack of response, we’re definitely working on this issue but a huge issue popped up and took our attention away for a bit. We hope to have an answer soon.

Thanks!

anonymous_user_7e4e2181 · April 27, 2016, 7:32am

the use of TMap is not applicable to JSON – the simpler TArray must be used.

and, to give you the closest “TMap-like” functionality – your JSON will need to look like:

{
  "MyMap": [
    { "key":"MyKey1", "value":{ "TestData1":0, "TestData2":"bliz" } },
    { "key":"MyKey2", "value":{ "TestData1":1, "TestData2":"blaz" } }
  ]
}

take a look at this SO post for an example of USTRUCTs to C++ accessing the elements of a sample JSON:

AFAICT, when using [ Engine/Source/Runtime/JsonUtilities/Private/JsonObjectConverter.cpp ] – the engine will NOT allow you to make up a “key” name “on-the-fly” – you need to define it so the engine can make the properties accessible as standard C++ structs (which makes it possible to setup and be used with Blueprints).

having said that, it is possible to make your own key:value pairs – but, you will need to use the more low level TJsonWriter and TJsonWriterFactory & TJsonReader and TJsonReaderFactory functions. you will need to hold the engine’s hand when you fill your own USTRUCT that may be using TMap…

and then, your custom JSON parser and writer can read something like this:

{
  "MyMap": [
    { "MyKey1", { "0", "bliz" } },
    { "MyKey2", { "1", "blaz" } }
  ]
}

hope this helps!

anonymous_user_33e74471 · April 27, 2016, 1:52pm

Thanks , I’ll definitely check those out.

The biggest reason I consider it a bug is that if you do:

FString json;
MyMapHoldingStruct mystruct;
FJsonObjectConverter::UStructToJsonObjectString(MyMapHoldingStruct ::StaticStruct(), mystruct, json, 0,0);
FJsonObjectConverter::JsonObjectStringToUStruct(json, mystruct, 0, 0);

The deserialization will actually fail at deserialization because the json isn’t correctly formatted instead of at serialization when it decides it can’t serialize the map or later in deserialization when it reaches the field that’s not json serializable. It makes a piece of json that looks something like:

"MyMap": "((MyKey, ))

which should be more like:

"MyMap": "((MyKey, ""))

That said, the TJson[dostuffer]s should be more what I need.

Thanks!

anonymous_user_7e4e2181 · April 28, 2016, 8:51pm

to close this:

from the cpp file above, in the function ConvertScalarJsonValueToUProperty(), you can see the following supported JSON types (which should look familiar):

UNumericProperty
UBoolProperty
UStrProperty
UArrayProperty
UTextProperty
UStructProperty

when these checks fail, it will try to determine it from CoreUObject – which does not return an unrecognized object as “” or even null – it leaves it blank.

the actual JSON looks like this:

{ "testMap": "((\"Test\", ))" } // note the right side is all a string...

but, you’re asking for something like this, yeah?

{ "testMap": "((\"Test\", null ))" }

CoreUObject is a system i will not be allowed to change to make it work like this…

you will need to proceed with making your custom JSON handler.

anonymous_user_33e74471 · April 28, 2016, 10:00pm

I think the bigger issue for me is that when the TMap hits another UStruct the ExportTextItem only gets output as a diff from the default struct, so if you have an identical ustruct to the default ustruct it will get popped out as nothing. Otherwise the json is able to deserialize even though the tmap is formatted as a string rather than something else.

The change I did was to say when the value gets serialized in UMapProperty::ExportTextItem if the string returned is empty print () instead of printing nothing.

Sorry, it’s been a while now since I put this in so the actual issue was a bit fuzzy, but I can see from my diffs that I replaced this:

ValueProp->ExportTextItem(ValueStr, PropData + MapLayout.ValueOffset, PropDefault + MapLayout.ValueOffset, Parent, PortFlags | PPF_Delimited, ExportRootScope);

with this:

FString ValueCache;
ValueProp->ExportTextItem(ValueCache, PropData + MapLayout.ValueOffset, PropDefault + MapLayout.ValueOffset, Parent, PortFlags | PPF_Delimited, ExportRootScope);

if (ValueCache.IsEmpty())
{
	ValueStr += TEXT("()");
}
else
{
	ValueStr += ValueCache;
}

anonymous_user_33e74471 · April 28, 2016, 10:03pm

Sorry for kind of jumping around, but it’s hard to remember all the issues I was having because it was a while ago. Please excuse my shotgun attempt at trying to verbalize what little I remember from the issue.

But I think your suggestion for using the jsonreaders/writers fits my problem way better anyway, because I think the structure of the struct might change over time and we’ll be able to handle changes to the format of our data that way

anonymous_user_7e4e2181 · April 29, 2016, 3:16pm

, your patch above has wonderfully narrowed in a possible solution.

i’ve added the fix to: [ Engine/Source/Runtime/CoreUObject/Private/UObject/PropertyStruct.cpp ] in function: UStructProperty::UStructProperty_ExportTextItem()

and added to the last if statement there with

else
	{
		ValueStr += TEXT("()");
	}

tested and no errors!

anonymous_user_33e74471 · April 29, 2016, 5:51pm

Awesome!

Thanks so much for your help guys!

edit: Not sure how best to mark this answered since it’s all in comments to the op <_<

anonymous_user_7e4e2181 · April 29, 2016, 9:16pm

give it to - i work with her and she’s the one who gave me this task