G,day Everyone,
So over the last couple of days, I have been creating a Inventory System that uses a DataTable as a centralized Inventory Database. I pretty much have the DataTable Database system in a alpha working state(not bug free or performance enhanced) functionality wise.
I have been working on a mechanic to allow a game designer(me) to dynamically pass nest-able(?) wildcard structures to a DataTable, in my case, it would be a DataTable for a Inventory Database. The reasoning I decided to give this type of system a go in developing are:
Unreal Engine does not have an inbuilt offline database system? (I could be wrong. I simply might not be aware)
I wanted a lazy and time efficient approach to adding, updating or removing item data/stats
I was up very late, tired as hell when I started this, and it became an obsession due to its complexity
I thought I would just run you guys through what I have at the moment, just in case this information might come in handy to someone else, as I know I’ve spent a ton of time searching for answers online, through the documentation, and through the source code.
Allowing a Wildcard Structure into a Blueprint Node
.h
UFUNCTION(BlueprintCallable, Category = "Inventory System | Data Table", CustomThunk, meta = (DataTable = "DataTable", CustomStructureParam = "Structure"))
static void InsertIntoDataTable(UDataTable* DataTable, UProperty* Structure);
DECLARE_FUNCTION(execInsertIntoDataTable)
{
//Grab the UDataTable Object from the Stack
P_GET_OBJECT(UDataTable, DataTable);
// Step into the stack
Stack.Step(Stack.Object, NULL);
// Grab the last property found when we walked the stack
// This does not contains the property value, only its type information
UStructProperty* StructureProperty = ExactCast<UStructProperty>(Stack.MostRecentProperty);
// Grab the base address where the struct actually stores its data
// This is where the property value is truly stored
void* StructurePtr = Stack.MostRecentPropertyAddress;
// We need this to wrap up the stack
P_FINISH;
//Now we have our data from the stack, lets internally update the data table
InternalInsertIntoDataTable(DataTable, StructureProperty, StructurePtr);
}
static void InternalInsertIntoDataTable(UDataTable* DataTable, UStructProperty* Structure, void* StructurePtr);
So what we are doing here is defining the Blueprint function and setting it to CustomThunk, which will stop Unreal Header Tool(UHT) from generating the functions boilerplate code.
Next, we declare the function body with the DECLARE_FUNCTION macro. Inside the function body, we grab the DataTable, then walk the stack and get the most recent UProperty via Stack.MostRecentProperty. This will only get the property type information.
Using Stack.MostRecentPropertyAddress, we get the memory address to the actual property data.
Then we call our internal function and parse the newly received data.
Manipulating the DataTable with Json
Now if some of you have delved into this topic, you would know in C++, UDataTable has this lovely little method called AddRow() which takes a structure that inherits from FTableRowBase. Now unfortunately, since we are pulling a structure from the Blueprint Virtual Machince, we cannot explicitly make our Virtual Machine structure inherit from a C++ structure(::StaticStruct()), so this renders AddRow worthless in this instance.
So a work around to this problem, apart from creating a new asset derived from UDataTable and adding Add,Remove,Update etc methods, you are able to get all the contents from the DataTable in a Json String. You are also able to create a table from a Json string, providing the DataTable Row Structure has been set first.
void UBlueprintFunctionsLibrary::InternalInsertIntoDataTable(UDataTable* DataTable, UStructProperty* Structure, void* StructurePtr)
{
//Get all the contents as a Json string. this Json string also has the Row Structure Information in it as well... pretty handy
FString TableDump = DataTable->GetTableAsJSON(EDataTableExportFlags::None);
//^ Will output a Json string like the following: {"Name": "RowName", "Var1" : "Var1Value"}] ---No master Identifier hence bellow comment
//A function to add "{\"Table\":" to the beginning and "}" to the end of our TableDump. The reason for this is that I am yet to find a function that allows me to read or write a Json string without an Identifier.
JsonAddTablePrefix(TableDump);
TSharedPtr<FJsonObject> JsonObject = MakeShareable(new FJsonObject()); //Create a master FJsonObject
TSharedRef<TJsonReader<>> JsonReader = TJsonReaderFactory<>::Create(TableDump); //Read the TableDump into the master FJsonObject for data manipulation
FString write; //Our Serialized Json string - Will be ready to use once we've completed our data manipulation and serialized our master FJsonObject
TSharedRef<TJsonWriter<wchar_t, TPrettyJsonPrintPolicy<wchar_t>>> JsonWriter = TJsonWriterFactory<wchar_t,TPrettyJsonPrintPolicy<wchar_t>>::Create(&write); //Our Writer Factory
if ( FJsonSerializer::Deserialize(JsonReader, JsonObject) && JsonObject.IsValid() ) //Deserialize the master FJsonObject
{
UE_LOG(InventorySystem, Warning, TEXT("JsonObject Deserialized!, proceeding..."));
TArray<TSharedPtr<FJsonValue>> Items = JsonObject->GetArrayField("Table"); //Read the field array (More information on Json https://www.json.org/json-en.html if you are unfamiliar with Json)
TSharedPtr<FJsonObject> NewItem = MakeShareable(new FJsonObject());
NewItem = JsonFromStructure(Structure, StructurePtr); //A function that iterates over our wildcard structure. Code bellow.
TSharedPtr<FJsonValueObject> NewJsonValue = MakeShareable(new FJsonValueObject( NewItem )); //Cast the FJsonObject NewItem to FJsonValueObject so it can be added to a Json field array
if(!NewJsonValue.IsValid())
{
UE_LOG(InventorySystem, Warning, TEXT("NewJsonValue not Valid!"));
return;
}
//Add our NewJsonValue FJsonValueObject the Items array, then add to the field array called "Table"
Items.Add(NewJsonValue);
JsonObject->SetArrayField("Table", Items);
{
//We have finished data manipulation and now we serialize our FJsonObject
FJsonSerializer::Serialize(JsonObject.ToSharedRef(), JsonWriter);
//For some reason, CreateTableFromJSONString will not accept Json Pretty Policy, it will only accept a single line JsonString, so the
//The SanitizeJsonString removes all the spaces(Will change to just removing new lines when I find a practical solution)
FString str = SanitizeJsonString(write, true);
//Update the DataTabe from the new Json string.
TArray<FString> errors = DataTable->CreateTableFromJSONString(str);
for ( auto e : errors )
{
UE_LOG(InventorySystem, Warning, TEXT("Error: %s"), *e);
}
}
}
I have comment the code for ease of understanding.
Iterating over a Blueprint VM Structure and getting values
Ok, so bellow, I have pasted the code for iterating over the wildcard structure and reading its variables and values. Note: This only allows for basic types such as FStrings, FNames, FText, bools, ints and floats as these are the only properties available to read. I suppose if you really require more types, you can delve deeper into the engine and find a way to create a new property type. See UnrealType.h for a better understanding on this.
TSharedPtr<FJsonObject> UBlueprintFunctionsLibrary::JsonFromStructure(UStructProperty* Structure, void* StructurePtr)
{
TSharedPtr<FJsonObject> JsonObject = MakeShareable(new FJsonObject());
if(!JsonObject.IsValid())
{
UE_LOG(InventorySystem, Warning, TEXT("JsonObject not Valid!"));
return nullptr;
}
JsonObject->SetStringField("Name", "Item");
JsonIterateStructure(Structure, StructurePtr, JsonObject);
return JsonObject;
}
void UBlueprintFunctionsLibrary::JsonIterateStructure(UStructProperty* Structure, void* StructurePtr, TSharedPtr<FJsonObject>& JsonObject)
{
// Walk the structs' properties
UScriptStruct* Struct = Structure->Struct;
for (TFieldIterator<UProperty> It(Struct); It; ++It)
{
UProperty* Property = *It;
// This is the variable name if you need it
FString VariableName = Property->GetAuthoredName();
// Never assume ArrayDim is always 1
for (int32 ArrayIndex = 0; ArrayIndex < Property->ArrayDim; ArrayIndex++)
{
// This grabs the pointer to where the property value is stored
void* ValuePtr = Property->ContainerPtrToValuePtr<void>(StructurePtr, ArrayIndex);
// Parse this property
JsonParseStructureProperty(Property, ValuePtr, JsonObject, VariableName);
}
}
}
void UBlueprintFunctionsLibrary::JsonParseStructureProperty(UProperty* Property, void* StructurePtr, TSharedPtr<FJsonObject>& JsonObject, FString VariableName)
{
// Here's how to read integer and float properties
if (UNumericProperty* NumericProperty = Cast<UNumericProperty>(Property))
{
if (NumericProperty->IsFloatingPoint())
{
JsonObject->SetNumberField(VariableName, NumericProperty->GetFloatingPointPropertyValue(StructurePtr));
}
else if (NumericProperty->IsInteger())
{
UE_LOG(InventorySystem, Warning, TEXT("Iterate Int"));
}
}
// How to read booleans
if (UBoolProperty* BoolProperty = Cast<UBoolProperty>(Property))
{
JsonObject->SetBoolField(VariableName, BoolProperty->GetPropertyValue(StructurePtr));
return;
}
// Reading names
if (UNameProperty* NameProperty = Cast<UNameProperty>(Property))
{
FName name = NameProperty->GetPropertyValue(StructurePtr);
if (name.ToString().Len() <= 0)
{
UE_LOG(InventorySystem, Warning, TEXT("FName not Valid!"));
return;
}
JsonObject->SetStringField(VariableName, name.ToString());
return;
}
// Reading strings
if (UStrProperty* StringProperty = Cast<UStrProperty>(Property))
{
FString string = StringProperty->GetPropertyValue(StructurePtr);
if (string.Len() <= 0)
{
UE_LOG(InventorySystem, Warning, TEXT("String not Valid!"));
return;
}
JsonObject->SetStringField(VariableName, string);
return;
}
// Reading texts
if (UTextProperty* TextProperty = Cast<UTextProperty>(Property))
{
FText text = TextProperty->GetPropertyValue(StructurePtr);
if (text.IsEmptyOrWhitespace())
{
UE_LOG(InventorySystem, Warning, TEXT("Text not Valid!"));
return;
}
JsonObject->SetStringField(VariableName, text.ToString());
return;
}
// Reading an array
if (UArrayProperty* ArrayProperty = Cast<UArrayProperty>(Property))
{
UE_LOG(InventorySystem, Warning, TEXT("Iterate Array"));
// We need the helper to get to the items of the array
FScriptArrayHelper Helper(ArrayProperty, StructurePtr);
TArray<TSharedPtr<FJsonValue>> ArrayToAdd;
for (int32 i = 0, n = Helper.Num(); i < n; ++i)
{
TSharedPtr<FJsonValue> Val = FJsonObjectConverter::UPropertyToJsonValue(Property, StructurePtr, NULL, NULL);
if (!Val.IsValid())
{
UE_LOG(InventorySystem, Warning, TEXT("Array Invalid!"));
return;
}
ArrayToAdd.Add(Val);
}
JsonObject->SetArrayField(VariableName, ArrayToAdd);
return;
}
// Reading a nested struct
if (UStructProperty* StructProperty = Cast<UStructProperty>(Property))
{
UE_LOG(InventorySystem, Warning, TEXT("Nested Object Valid!"));
TSharedPtr<FJsonObject> StructObject = MakeShareable(new FJsonObject());
if (!StructObject.IsValid())
{
UE_LOG(InventorySystem, Warning, TEXT("Nested Object not Valid!"));
return;
}
JsonIterateStructure(StructProperty, StructurePtr, StructObject);
JsonObject->SetObjectField(VariableName, StructObject);
return;
}
else
{
UE_LOG(InventorySystem, Warning, TEXT("Nested Object not Valid! %s"), *Property->GetAuthoredName());
}
}
This last piece is pretty self explanatory. We read over the properties and if there is a nested structure, we recursively iterate over nested structures and their properties until we finish the iterations. We add these values to the master FJsonObject via reference.
Question Time:
-
Does anyone know if there is a function to in Unreal Engine to convert a Pretty Json string to a Condensed Json string?- I found the solution while editing this post. See below for more information. - Does anyone know of a method to make a VM structure: UStruct, UStructProperty, UScriptStruct inherit from a C++ UStruct such as FTableRowBase?
- Has anyone created their own UProperty type? e.g UStrProperty
Converting from Pretty Json to Condensed Json and vice versa
This can be achieved by our Writer Factory when we are creating it. Unreal Engine has 2 Json style policies.
TPrettyJsonPrintPolicy
TCondensedJsonPrintPolicy
To include them, the hear files are located in runtime\Json\Public\Policies
//For condensed Json strings
TSharedRef<TJsonWriter<wchar_t, TCondensedJsonPrintPolicy<wchar_t>>> JsonWriter = TJsonWriterFactory<wchar_t,TCondensedJsonPrintPolicy<wchar_t>>::Create(&write);
//For pretty Json strings
TSharedRef<TJsonWriter<wchar_t, TPrettyJsonPrintPolicy<wchar_t>>> JsonWriter = TJsonWriterFactory<wchar_t,TPrettyJsonPrintPolicy<wchar_t>>::Create(&write);
Notes and Sources
I will provide some notes on some issues that I have come across while R&Ding this.
- Don’t forget to add Json and JsonUtilities to the .Build.cs - Double check… If you use anything inside a folder in Runtime, it has to be added as a module.
- JSON
- Unreal Engine API Reference
- Engine Source Code
- [Tutorial] How to accept wildcard structs in your UFUNCTIONs - Community Content, Tools and Tutorials - Unreal Engine Forums
- https://forums.unrealengine.com/development-discussion/c-gameplay-programming/1363884-let-s-talk-json