Creating struct/class instances from json payload

Hey guys,
I’ve been trying to wrap my head around this for a while and I can’t come up with a generic solution and my stubborn self isn’t satisfied with a hardcoded one.

Consider the following case: (abbreviated for clarity, hopefully)

// Stuff that doesn't change per item (type). doesn't need to be serialized
struct FItemBaseData : TableRowBase
{
FString Name;
Texture2d Icon
}

// custom data that accompanies a specific item
// could be any selection of data for different item types
class UStackableItemData : UItemData
{
int32 Quantity;
}

class UWeaponItemData : UItemData
{
int32 AmmoCount;
}

struct FInventoryItem
{
FItemBaseData BaseData;
UItemData ItemData;
}

class UInventoryComponent : UActorComponent
{
TArray<FInventoryItem> Items;
void RequestItemsFromDB();
}

The use-case is to de/serialize the UItemData to and from a database.
As I imagine it the the whole Items array would arrive in one payload. Without knowing better I’d assume a single data request is probably better to manage and more efficient in the long run.

Now, several problems arise:

  1. The FJsonObjectConverter can only convert UStructs to JSON payload and back again. It seems to do that neatly though. But UStructs, while you can derive, them you cannot down-cast them without losing the data from its children.
  2. That’s why I wanted to use UObjects instead because you can cast them as desired. But then you lose the ability to neatly serialize/de-serialize them into JSON
  3. Even if serialization works I didn’t find a way to create the right instance from a payload without manually checking what kind of payload it is (by passing meta data in the payload for instance). So I cannot generically create an object/struct instance from a payload.

E.g I’d like to avoid doing

UItemData* GetItemData(payload)
switch (payload.customdataclass)
{
case "Stackable":
UStackableItemData* data;
JsonToUObject(data, payload);
return data;

case "Weapon"
UWeaponItemData* data;
JsonToUObject(data, payload);
return data;
[...]
}
}

Is there a generic solution that I’m overlooking? Is there a different approach? Am I overthinking this? Am I really just too stubborn?

Thanks!

Allow me to introduce you to the answer to all your problems (and, potentially, the source of entirely new and different problems): REFLECTION!

(No, not that type of reflection, the other type.)

Off the top of my head, and with the caveat that this is wildly oversimplified, but as a general example…

  1. For each item in your list, obtain the matching UClass* or UStruct* with FindObject<>(); to make this easier, I might actually just make a data asset or something with a list of item type → class mappings, and then look up the appropriate thing to instanciate via that, rather than having to use FindObject.
  2. Create a new instance of said class/struct with NewObject()
  3. For each field in the JSON record for that particular item, use UKismetSystemLibrary::SetStructurePropertyByName on the newly instanced class/struct to set the matching UPROPERTY() (or some other method of reflection).
  4. Do something with the new properly-set-up instance of the thing. (Store it in an inventory component, print it out and turn it into origami, whatever.)
  5. *jazzhands and/or victory fanfare*

(Someday I will stop answering questions on here in an irreverent tone at 2am when I can’t sleep. Evidently, today is not that day.)

Hey @Packetdancer,
That’s some good food for thought, but definitely not easy to digest :smiley:

I’ll need to think my approach over once again and play around with it. The victory fanfare was definitely somewhat premature :stuck_out_tongue:

I think I’ll go with a struct layout since this is definitely more digestible and way easier to serialize and seems more reasonable since this is handling pure data and no logic. How I’m gonna manage those is definitely up for debate though. Currently toying with the idea of an arbitrary property map system… Something like

struct FSerializeableProperties
{
TMap<FName, int32> IntProperties;
[...]
}
struct FItemData : FSerializeableProperty
{

}

// use this somewhere
ItemData.AddProperty(FName("Quantity"), Item.Quantity);

Dunno, exactly yet. Doesn’t seem great either. Mulling it over. Way too many options

I definitely wouldn’t go with implementing your own arbitrary property map system, since you’d be more or less duplicating the existing reflection functionality of Unreal’s own UPROPERTY system; that seems like wasted effort to me, especially since if you use the existing system you get the benefit that all the properties are exposed and accessible in the way that the editor and all other tools expect. And that (having them part of the existing property system) has a ton of benefits, like the fact that you can natively access your own properties in Blueprint as you would with any other Unreal property; it’s a lot cleaner to just get the Ammo value off of a weapon rather than having to go through your own custom serialization system.

The reflection technique is literally just how you access that system via the property names rather than the C++ class/struct properties behind them.

That said, I rather forgot last night that at least for some simple cases, there’s code that’ll do this for you: namely, FJsonObjectConvertor. Beyond that, you can make some even more flexible methods of loading object definitions if you play with reflection in general more; there’s an old (but still mostly relevant) blog post on Unreal’s reflection system on their blog here.

You can also check out JsonAttributesToUStructWithContainer inside of Engine/Source/Runtime/JsonUtilities/Private/JsonObjectConverter.cpp to see an example of how you can iterate across the properties of a given class or structure (as ContainerPtr instances) and then get a ValuePtr for a given property on a given target object, and use that to set the value.

Just to clarify, the arbitrary property map wouldn’t be for the serialization per se. It’s more of the idea of managing data within the DataBase (and also the unreal front-end). I’m not sure this is a good idea either to store JSON blobs with arbitrary properties but it would enable me to reduce maintenance and manual extension for new item types which have new/different properties. I’d just get the blob and the actual object then handles it.

Something like that as opposed to having a different table for each item type

It introduces a lot of redundant data though and it doesn’t even catch all cases, think enums, (which would need to be converted to string beforehand) and other custom data, and I’m struggling to consolidate all the data bits everywhere. Ah heck anyways. I’d still use the FJsonObjectConverter on the PropertyMap since that one is working super well.

Essentially my goal is to create a workflow/system to eliminate as much manual intervention as much as possible (especially on the database side of things), including boiler plate code for each new persistent type added to the project. Doesn’t help that most of the topics are out of my comfort zone and experience while juggling 2-3 ideas at a time.

Fun aside: In a previous attempt we started to use the ustruct definitions to generate the database tables and relations, which sorta worked, but it didn’t really make things that much simpler :see_no_evil:

Your input has been of great value of prodding me into the right direction though. I feel like I’m moving into the right direction (or at least moving somewhere) :smiley:

Glad you feel like progress is happening!

I actually did something a little like this before, where given a table of type ID to class, it could just create and populate a tree of all the appropriate classes.

(I half-wonder if it would be worth redoing that system as a UE5 plugin and throwing it up onto GitHub…)

1 Like

I for one would be interested to at least take a gander.

I just realized something, and there must be a way to do it, but I don’t see it:

You can generate a class instance dynamically from its UClass

	UClass* ClassType = ItemTest->GetClass();
	auto* GeneratedItem = NewObject<UObject>(this, ClassType);

Is there an equivalent for structs? That way I wouldn’t need to store arbitrary data blobs, but can supply the UStruct data directly and generate my struct instance to deserialize. That could help quite a bit.

auto GeneratedStruct = GetStructForItem(JSON);
FJsonObjectConverter::JsonObjectStringToUStruct(JSON, &GeneratedStruct);

:thinking:

What your’e hitting on right now is more or less what I was trying to suggest! If you can instanciate the appropriate class (or struct) by doing a lookup – “this name for an object type maps to this UClass/UStruct” – and then using reflection to just go “okay, it has this ‘Name’ property and I have a ‘name’ field in this JSON object, so I’ll set that value” and so on. (Or using the aforementioned FJsonObjectConvertor.)

The end goal being that you never need to change the main code, just add a new map for “type name” → actual UClass/UStruct any time you added a new type of thing to the database.

That said, I have not experimented with creating UStruct instances on the fly. I think you would basically just allocate memory (UStruct::GetStructureSize() to get how much you need to allocate) and then cast it to what you need, but take that with a large grain of salt.

I didn’t find a way to instantiate what I need at runtime. I can use NewObject<UItemProxy>(UClass) just fine, but then its just an UItemProxy, which doesn’t know the members of UItemStackableProxy for instance. I didn’t find a way to downcast to UItemStackableProxy from the UClass either (I don’t think it is possible even?) or to its actual type without hardcoding it to Cast<UItemStackableProxy>, leading to a bunch of boilerplate for each item type I want to avoid and which is not great to maintain.
Also I’m hesitant to resort to reflection. Seems messy and high on maintenance, in addition to higher code complexity.

However I did make some progress from this point. If this is the data layout

USTRUCT()
struct FStackableItemData
{
	GENERATED_BODY()
		UPROPERTY()
		int32 Quantity;
};

USTRUCT()
struct FWeaponItemData
{
	GENERATED_BODY()
		UPROPERTY()
		float Durability;
};

and this is the class layout

UCLASS()
class UItemProxy : public UObject
{
	GENERATED_BODY()
public:
	virtual void Load(const FString& Payload)
	{
		// throw not implemented
	}
	virtual void Save(FString& Payload)
	{
		// throw not implemented
	}
};

UCLASS()
class UStackableItemProxy : public UItemProxy
{
	GENERATED_BODY()
public:
	FStackableItemData ItemData;

	virtual void Load(const FString& Payload) override
	{
		FJsonObjectConverter::JsonObjectStringToUStruct(Payload, &ItemData, 0, 0);
	}

	virtual void Save(FString& Payload) override 
	{
		FJsonObjectConverter::UStructToJsonObjectString(ItemData, Payload); 
	}
};

I can iterate the TArray<UItemProxy> on the inventory and call Save([...]) to receive the correct json payload from FStackableItemData and attach the UClass to store it away for later use.
When receiving data from the backend I can create NewObject<UItemProxy>(UClass) and then .Load([...]) to populate the FStackableItemData struct accordingly.

I also reduced the boilerplate here into a macro so creating new item types is straight forward with few possibilities of error:

#define SAVELOAD(ustruct) \
		virtual void Load(const FString& Payload) override \
		{ FJsonObjectConverter::JsonObjectStringToUStruct(Payload, &ustruct, 0, 0); } \
		virtual void Save(FString& Payload) override \
		{ FJsonObjectConverter::UStructToJsonObjectString(ustruct, Payload);}

UCLASS()
class UWeaponItemProxy : public UItemProxy
{
	GENERATED_BODY()
public :
	FWeaponItemData ItemData;
	SAVELOAD(ItemData)
};

Not the most ideal solution, but it is functional and should serve my purposes for the time being. I would think a more seasoned cpp programmer might raise an eyebrow, but hey. first it needs to function, then it can get gud :smiley:

Hypothetically, if you’re using classes instead of structs, I believe you can:

  1. Get the UClass* (from a lookup table or whatever).
  2. Create it as the generic parent class of your item (UMyItemBase or whatever), but using that UClass; while the resulting class will be cast to a UMyItemBase*, the underlying object will still be a UWeaponItem or whatever subclass that UClass* represented.
  3. Walk the class definition (i.e., that same UClass*) to get the properties if you want to set them by hand, or just pass the object to the FJsonObjectConverter.
  4. Now you have a whatever-it-was, but cast to the generic item (and thus suitable for storing right into an inventory or whatever else).