Runtime DataTable - Import text CSV or Google Sheet at runtime and fill an array of structs!

[Jared Therriault]

Would it be possible to export a data structure to excell in runtime?

Hi danibb - not with this plugin at this time, but this is a request I see a lot. It is something I will consider for the future.

[AaronMTodd]

Would it be possible to export a data structure to excell in runtime?

I second this request! I have some complex structs that would be great to export and be able to manipulate. I'm sure I read somewhere that this supports structs within structs, correct? It would be nice to be able to export just to make sure that a complex CSV like that is setup correctly for import.

[Jared Therriault]

I second this request! I have some complex structs that would be great to export and be able to manipulate. I'm sure I read somewhere that this supports structs within structs, correct? It would be nice to be able to export just to make sure that a complex CSV like that is setup correctly for import.

That's a fair use case, thank you for the feedback! It is true, the plugin supports structs within structs. For now, I'd recommend creating a normal DataTable in the engine with a nested struct then exporting it to CSV to get a feel for the layout, then create new rows in Google sheets based on that.

The heat is rising on this feature request, but I'll be honest: This is something I've looked into and the amount of work for implementation wouldn't be trivial. So if I do make this feature happen it will be part of a separate plugin. Reasons being:

• I don't want to raise the price of RDT as it is
• People who don't need the export feature might feel burned having to pay more when it didn't use to be like that
• The time investment for implementation, testing and support will far surpass the amount of money I make from the plugin to begin with

So it would be a separate buy-in. I'm not sure how customers would react to that. How do you feel?

[joesys]

Hi Jared,

Thank you for the wonderful plugin.

I notice the following code in ReadCSVRow()

//Ensure we won't end up with an index under 0 (in case of an empty string)
int32 FieldIndex = (fields[CSV_FieldIndex].Len() - 1 < 0 ? 0 : fields[CSV_FieldIndex].Len() - 1); //todo: keep an eye on this

if (fields[CSV_FieldIndex].Len() > 1 && fields[CSV_FieldIndex][FieldIndex] == '\\') //if we find a wrapper character and the last character was supposed to escape it then it was obviously included by the user
{
    fields[CSV_FieldIndex][FieldIndex] = WrapperCharacter;
    StateIndex = 3; //Quote within user string/text
}

StateIndex = 2; //Wrapped wrapper

I believe it should be:

...

    StateIndex = 3; //Quote within user string/text
}
else 
{
    StateIndex = 2; //Wrapped wrapper
}

Otherwise, StateIndex will always be 2.

[Crusader_91]

Hello, Jared!

I have encountered an issue working with your plugin.

It doesn't encode cyrillic symbols correctly after import from Google spreadsheets. For example, russian word "Приказ" was encoded to "?@8:07". I think, that's because of default UTF-16 encoding. Is it possible to use UTF-8 encoding during import?

[Jared Therriault]

Hello, Jared!

I have encountered an issue working with your plugin.

It doesn't encode cyrillic symbols correctly after import from Google spreadsheets. For example, russian word "Приказ" was encoded to "?@8:07". I think, that's because of default UTF-16 encoding. Is it possible to use UTF-8 encoding during import?

Hi Crusader_91, maybe you can help me figure out when this is happening. After 'Call On Complete' is called, save the payload and send that to me please!

I'm currently using wchar_t to parse the payload into structs and it's my understanding that should support Cyrillic, so I would like to know if the garbage data is coming from Google or if the script converts it to garbage. Send some samples when you can! Thank you.

[Crusader_91]

Hello Jared,

This is a duplicate of the letter I sent you personally.

We were talking about an issue about encoding cyrillic symbols in the downloaded information. You asked me to send the payload string to you. Here it is (partially):


========================================================================

LogBlueprintUserMessages: [BP_ItemManager_2] ---,Item Name: Identified,Item Class,Item Type,Item Mesh In World,Item Materials,Item UI Image,Description,Usable,Equipable,Stackable,Max Stack Count,Item Price,OneItemMass,PivotPointShiftInItemInspector,EventsToActivateOnPickup
00000000,"(RU=""Масло навыка: Перехват"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_Intercept.BP_Equip_Oil_Skill_General_Intercept_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/MI_ItemsUI_Oil_Skill_Gener
al_Intercept.MI_ItemsUI_Oil_Skill_General_Intercept,"(RU=""Капсула с Маслом, дающая персонажу навык: Перехват"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
00000001,"(RU=""Масло навыка: Дезориентирующий приказ"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_DisorientOrder.BP_Equip_Oil_Skill_General_DisorientOrder_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/M
I_ItemsUI_Oil_Skill_General_Disorient.MI_ItemsUI_Oil_Skill_General_Disorient,"(RU=""Капсула с Маслом, дающая персонажу навык: Дезориентирующий приказ"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None

========================================================================

As I can see, the payload string is correct, so the garbage data is definitely not coming from Google. Each segment of the Russian-language text (after the labels "RU") is reproduced correctly.

I also attach a link to a copy of this table, so you can make sure the information is correct:

https://docs.google.com/spreadsheets...it?usp=sharing

Thanks for your work again!

[Jared Therriault]

Hi Jared,

Thank you for the wonderful plugin.

I notice the following code in ReadCSVRow()

//Ensure we won't end up with an index under 0 (in case of an empty string)
int32 FieldIndex = (fields[CSV_FieldIndex].Len() - 1 < 0 ? 0 : fields[CSV_FieldIndex].Len() - 1); //todo: keep an eye on this

if (fields[CSV_FieldIndex].Len() > 1 && fields[CSV_FieldIndex][FieldIndex] == '\\') //if we find a wrapper character and the last character was supposed to escape it then it was obviously included by the user
{
fields[CSV_FieldIndex][FieldIndex] = WrapperCharacter;
StateIndex = 3; //Quote within user string/text
}

StateIndex = 2; //Wrapped wrapper

I believe it should be:

...

StateIndex = 3; //Quote within user string/text
}
else
{
StateIndex = 2; //Wrapped wrapper
}

Otherwise, StateIndex will always be 2.

You're right joesys! I actually saw this oversight a few weeks ago and fixed it localy but I haven't pushed the fix. I will be pushing a fix soon (in the next month). This should only affect users who have double-nested quotes in their text and heavy use of "\". For most users this won't be an issue, but a bug is a bug and I'll have the fix out soon! Thank you!

[Jared Therriault]

Hello Jared,

This is a duplicate of the letter I sent you personally.

We were talking about an issue about encoding cyrillic symbols in the downloaded information. You asked me to send the payload string to you. Here it is (partially):


========================================================================

LogBlueprintUserMessages: [BP_ItemManager_2] ---,Item Name: Identified,Item Class,Item Type,Item Mesh In World,Item Materials,Item UI Image,Description,Usable,Equipable,Stackable,Max Stack Count,Item Price,OneItemMass,PivotPointShiftInItemInspector,EventsToActivateOnPickup
00000000,"(RU=""Масло навыка: Перехват"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_Intercept.BP_Equip_Oil_Skill_General_Intercept_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/MI_ItemsUI_Oil_Skill_Gener
al_Intercept.MI_ItemsUI_Oil_Skill_General_Intercept,"(RU=""Капсула с Маслом, дающая персонажу навык: Перехват"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
00000001,"(RU=""Масло навыка: Дезориентирующий приказ"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_DisorientOrder.BP_Equip_Oil_Skill_General_DisorientOrder_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/M
I_ItemsUI_Oil_Skill_General_Disorient.MI_ItemsUI_Oil_Skill_General_Disorient,"(RU=""Капсула с Маслом, дающая персонажу навык: Дезориентирующий приказ"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None

========================================================================

As I can see, the payload string is correct, so the garbage data is definitely not coming from Google. Each segment of the Russian-language text (after the labels "RU") is reproduced correctly.

I also attach a link to a copy of this table, so you can make sure the information is correct:

https://docs.google.com/spreadsheets...it?usp=sharing

Thanks for your work again!

Taking this 1-to-1 so I can ask about project-specific information.

[Jared Therriault]

Taking this 1-to-1 so I can ask about project-specific information.

For those reading, we worked this out. For some reason the marketplace gave this an outdated version of the plugin - if you're on 4.24 and experience issues with non-latin writing systems when using this plugin let me know.

[IgnacioFarias]

Hi! Im developing for Android and Im not sure what I should use as the Save/Load Path. Im also using this plugin to download images from urls and save them, would I use this same path for that?

Thanks!

[Jared Therriault]

Hi! Im developing for Android and Im not sure what I should use as the Save/Load Path. Im also using this plugin to download images from urls and save them, would I use this same path for that?

Thanks!

Hello IgnacioFarias,

Save Path and Load Path are optional provided you're using an online Google Sheet. If you provide a path, the plugin will save the Google Sheet in CSV format. You can then load the CSV from the path in the event your game is offline or you don't provide a Google Sheets URL.

If you want to provide a path, then the easiest method on mobile is to use relative paths through blueprint. Searching for "Paths" in blueprint nodes, you can build something like this:



UE4 will return the appropriate path based on the operating system this way, then the plugin will construct any missing folders as needed.

As for your second question, this plugin won't download images directly. I assume that you're storing image URLs in a spreadsheet and downloading them separately, correct? If that's the case then you can save them anywhere that makes sense to you. The plugin doesn't have anything to do with that But as a general tip, I'd still recommend using the Paths blueprint nodes just for ease of use.

[Jared Therriault]

Oct 14 2020

Hey everyone, couple of updates!

First, there was an issue with my 4.24 fix that was supposed to fix support for accented characters. This issue has been fixed now and is live. Fixing my fix

Second, on the 4.25+ version:

File Update and Price Increase incoming!

Coming in the next couple of weeks:

1. Update Object variables directly as an alternative to creating structs (though structs are still an option)
2. Variable order no longer matters, now matching by header name (or do it the old way if you prefer)
3. Local CSV Export from structs or objects - you can even choose which variables get exported or skipped. Thank you everyone for your requests!
4. Inclusion of easyCSV - this update includes a massive refactor of CSV parsing code which is much faster and reusable. If using a local CSV, it's also construction script safe. easyCSV is now a separate plugin embedded into Runtime DataTable.
5. $15 USD will be the new price, so get it now to lock in the current price!

Q&A:

Why are you raising the price?

Getting export working was not trivial. It involved synching up import and export functions to read and write data the exact same way depending on the data type itself and also be compatible with the way the engine exports data as string, so it involved a lot of trial and error and testing. Because of the amount of time it took, I thought it would be reasonable to ask for a little more than before rather than making it a separate plugin. This way all existing owners get the update for free. Thank you for being patient!

In addition to export functionality, I'm polishing up a new feature that will affect variables on objects directly. No need to create structs anymore, though structs will of course still work. I was getting bothered by the cumbersome nature of having to add more members to my struct every time I wanted to link it to my Google Sheet. My preferred workflow is just to right-click a pin and promote it to variable, not go out of my way to update a struct and break it wherever I need it. You can still update structs the same way, but now you can update object variables directly too.

Of course, this required a new method of populating variables from the sheet. Structs can be filled sequentially - the sequence of members in a class is not so clear-cut. A sequence exists, but is not easily apparent to the end user. So I decided instead to match the header names with the variable names. This is the only way to update object variables directly, so be sure to make your headers and variable names match if you want to go this route. If using a struct, you have the option of a sequential or name-matching update. If you don't want to have to update anything, just use sequential (default).

Finally, I broke out the CSV parsing functionality into its own plugin. It's intended as a lower-cost alternative to RDT, for people dealing only in strings or those that don't mind converting the data on their own. It has no Google Sheets functionality. easyCSV, as I call it, will be available on the marketplace soon, but it is included with RDT for free in full. It is a faster parser that also supports new lines, commas and quotes inside quotes. It's just a much more robust solution which was necessary to develop to accommodate local export.

I hope this added value feels worth it to potential buyers!

Local Export? Does that mean there won't be Google Sheets export?

That's correct, export is local-only. Google Sheets allows anonymous users to read public sheets, but you can't write anonymously to sheets at all through the API even if the sheet is public - you must authorize yourself. Thus, I decided not to include that functionality in this release. Reading public Google Sheets will work the same as it always has. No functionality has been removed from this release, only added.

Will Google Sheets export come later?

I am working on OAuth support for RDT, but it will come as a separate plugin. After my exploration of the idea, I found that the effort required to implement it would be pretty hefty. Given that knowledge, I decided that hiking the price up even further for this feature that most wouldn't use would be beneficial to the largest group, so when I pursue private sheets and exporting to sheets it will be for a separate product that makes OAuth its primary focus.

Will these updates come to 4.24 or earlier?

No, the only reason being that 4.25 introduced a major structural change in how properties are accessed which required a major refactor of the code. Retrofitting this into the 4.24 implementation would be a huge undertaking and I'm not sure all functionality would even work in 4.24, again because of the way properties are accessed. This new update will only be for 4.25 and newer.

Will my workflow change?

Yes and no. There will be some structural changes but won't affect everyone, and those who are affected will have until the next release to update their code. The full list of deprecated functions will be in the changelog when this update releases and the documentation will be updated to reflect the new changes as well. If all you do is fill structs with the basic node, you don't need to worry about code changes. This will only affect those using the RuntimeDataTableActor - you'll need to get a new struct inside of the Actor to get information like keys and headers or access cell data.

[Jared Therriault]

Hi everyone! The Big Island update has been pushed to the marketplace, now we're just waiting on Epic approval! Keep in mind that the price will increase to $15 USD when the update is approved, so this is your last chance to pick it up at the current price!

Highlights:

• Update Object variables directly as an alternative to creating structs
• Variable order no longer matters, now matching by header name (or do it the old way if you prefer)
• Local CSV Export from structs or objects - you can even choose which variables get exported or skipped
• Inclusion of easyCSV - this update includes a massive refactor of CSV parsing code which is much faster and reusable. If using a local CSV, it's also construction script safe.

Full Changelog:
("Big Island" Update, Nov 1 2020):
-NEW: Added Construction Safe version of the node that does not support Google Sheets but can be called from the construction script. It's called "UpdateArrayFromCSV_Info()" and updates the array input values on the blocking game thread from an FCSV_Info struct which can be generated using "MakeCSV_InfoFromString()" or "MakeCSV_InfoFromFile()".
-NEW: Added functionality to update individual UObjects now. Any class values can be filled just like a struct array. All wildcard struct inputs now accept struct arrays or UObject arrays.
-NEW: A new module has been added, easyCSV. This is a faster implementation of the CSV parsing that is less prone to error and also supports special characters inside of quotes preceded by "". For instance, you can quote inside quotes with " or create a new line with \n. This module will appear on the marketplace as a separate plugin, but will be included for all users of Runtime DataTable in full.
-NEW: Export functionality! Export does not support Google Sheets in this plugin, but it is now possible to export structs and object data to a new local spreadsheet! Node is called "ExportArrayToString". This CSV data as a string can then be saved to a local file or fed directly into a new data table. In C++, use GenerateCSV_FromArray().
-NEW: Added New Parameter to UpdateObjectOrStructArrayFromPublicGoogleSheetsOrLocalCSV() and UpdateArrayFromGoogleSheetsCSV_Internal() named "MatchStructMemberNames" that when true will attempt to match column names in your CSV with variables inside of your struct. This makes it so you don't have to have all variables in your struct represented sequentially in your CSV file. Note, name matching is slower than sequential updates so when working with very large data sets updates could take sometime longer to complete. In most cases however you shouldn't notice any speed difference. This parameter has no effect when using an array of objects because objects will always use name matching. Names must match exactly in terms of characters used, but capitalization does not matter. When false, header names in the CSV don't need to match, just the order of the variables.
-NEW: Some blueprint nodes now have less-often-used pins hidden behind a folding arrow. If you need them, just unfold the node.
-BUGFIX: Nodes that do support Google Sheets are now marked as 'UnsafeDuringActorConstruction' so they will not be available in the Construction Script. If you call a Custom Event from the Construction Script that calls a node that is unsafe, the plugin will print an error to the screen in development builds and in editor.
-BUGFIX: Made all nodes read an empty "OwningObject" pin as "Self" in blueprints. It no longer needs to be connected if the blueprint you're calling from owns the struct array or if the array is full of objects.
-INFO: "Struct Array To Fill" pin on the UpdateObjectOrStructArrayFromPublicGoogleSheetsOrLocalCSV() node has been renamed as "Array To Update" to be clearer as it can accommodate struct arrays or object arrays now. You may need to update your node connections in blueprint.
-INFO: In C++, the main method is now called UpdateArrayFromGoogleSheetsCSV_Internal(). The old method has been deprecated, so be sure to update your code. The new C++ method also has reordered parameters to allow for default parameters.
-INFO: The following functions have been deprecated in the the DynamicDataTable module and have been moved to the included easyCSV module. Please update your references for the next release as they will be removed. The new versions of these functions require an FCSV_Info input which is a public variable saved to the Runtime DataTable Actor or can be generated using "MakeCSV_InfoFromString()" or "MakeCSV_InfoFromFile()":

• SaveStringToFile
• LoadStringFromFile
• GetColumnAsStringArray
• GetRowValueAsString
• GetMapKeys
• GetMapKeyIndex

[Jared Therriault]

Update for 4.26 has been uploaded to Marketplace, now we wait on Epic response!