Announcement

Collapse
No announcement yet.

Runtime DataTable - Import text CSV or Google Sheet at runtime and fill an array of structs!

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • replied
    Originally posted by Studiotemp View Post
    Hi Jared,
    thanks for the nice plugin,

    just a simple question from a noob:
    I have developed an application running on a streaming publishing platform service, furioos.
    So I do not have any access to the files generated locally from your blueprint nodes.
    They won't help me on collecting the analytics text files generated from character location.

    So the question is: is there any way to write text files directly to google drive or anywhere else? I mean by changing the In Directory Path to a remote ftp path for instance??

    Thanks
    Click image for larger version

Name:	Untitled-1.jpg
Views:	24
Size:	191.2 KB
ID:	1865896
    Hi Studiotemp!

    I'm afraid Google doesn't allow anonymous API write access to Google sheets at all. Regardless of the access permissions on the sheet, only JWT-authorized applications can write to a sheet, and it's not trivial to do. You'll need to create a service account with Google (it's free) and authorize your game with a JWT handshake with your service account in C++ then rebuild the CSV file in JSON format to be accepted by Google's API as sheet data.

    I am working on a separate plugin for this that's just a generic way of working with Google Sheets, but it's far from complete. In the meantime, I would suggest bypassing all that and using an FTP or email plugin to send the exported CSV to yourself. If you need to upload that exported CSV to Google Sheets, you can do so through sheets script. If you need help with the script for that, I can send you some example code I wrote a few years ago that monitors a Google email address for messages with a few keywords then looks for a csv attachment, downloads it to Google Drive then imports it to the spreadsheet in question.

    To be clear, this functionality will probably not make its way into the base Runtime Data Table plugin. The time investment to create it is frankly huge. It would almost 100% surely be a separate plugin unless it ends up becoming a 'free-for-the-month' thing in which case the lump sum would justify adding all of that to the base plugin.

    Leave a comment:


  • replied
    Hi Jared,
    thanks for the nice plugin,

    just a simple question from a noob:
    I have developed an application running on a streaming publishing platform service, furioos.
    So I do not have any access to the files generated locally from your blueprint nodes.
    They won't help me on collecting the analytics text files generated from character location.

    So the question is: is there any way to write text files directly to google drive or anywhere else? I mean by changing the In Directory Path to a remote ftp path for instance??

    Thanks
    Click image for larger version

Name:	Untitled-1.jpg
Views:	24
Size:	191.2 KB
ID:	1865896

    Leave a comment:


  • replied
    Originally posted by MysticXSab View Post
    In my data table struct, I've put an actor class reference. Exporting the data table to CSV file and loading at runtime, the actor class reference is properly initialized. However, when I save out the CSV at runtime, whatever is being saved is not properly storing the actor class reference and I get an import error about that specific actor class reference on each row. Can you help me figure out how to properly save off the actor class reference?

    It should be: "BlueprintGeneratedClass'/Game/Pawn_PID.Pawn_PID_C'"
    But it comes out: "ClassProperty /Game/GainTest_Profile.GainTest_Profile:spawnSelection_121_79AB237D4B4989448F0B8799B1780D1C"

    --------------------------------------
    Update: It's all fine. I was using a regular Actor Class Reference variable and I needed to use Actor Class Soft Reference and resolve nodes where necessary.
    Glad you figured it out! That's what I would have advised - blueprint classes are created via internal construction script, so a hard reference will look like garbage data when it's serialized.

    Soft path is almost always the way to go. Cheers!

    Leave a comment:


  • replied
    In my data table struct, I've put an actor class reference. Exporting the data table to CSV file and loading at runtime, the actor class reference is properly initialized. However, when I save out the CSV at runtime, whatever is being saved is not properly storing the actor class reference and I get an import error about that specific actor class reference on each row. Can you help me figure out how to properly save off the actor class reference?

    It should be: "BlueprintGeneratedClass'/Game/Pawn_PID.Pawn_PID_C'"
    But it comes out: "ClassProperty /Game/GainTest_Profile.GainTest_Profile:spawnSelection_121_79AB237D4B4989448F0B8799B1780D1C"

    --------------------------------------
    Update: It's all fine. I was using a regular Actor Class Reference variable and I needed to use Actor Class Soft Reference and resolve nodes where necessary.
    Last edited by MysticXSab; 12-18-2020, 08:35 PM.

    Leave a comment:


  • replied
    Update for 4.26 has been uploaded to Marketplace, now we wait on Epic response!

    Leave a comment:


  • replied
    Hi everyone! The Big Island update has been pushed to the marketplace, now we're just waiting on Epic approval! Keep in mind that the price will increase to $15 USD when the update is approved, so this is your last chance to pick it up at the current price!

    Highlights:
    • Update Object variables directly as an alternative to creating structs
    • Variable order no longer matters, now matching by header name (or do it the old way if you prefer)
    • Local CSV Export from structs or objects - you can even choose which variables get exported or skipped
    • Inclusion of easyCSV - this update includes a massive refactor of CSV parsing code which is much faster and reusable. If using a local CSV, it's also construction script safe.

    Full Changelog:
    ("Big Island" Update, Nov 1 2020):
    -NEW: Added Construction Safe version of the node that does not support Google Sheets but can be called from the construction script. It's called "UpdateArrayFromCSV_Info()" and updates the array input values on the blocking game thread from an FCSV_Info struct which can be generated using "MakeCSV_InfoFromString()" or "MakeCSV_InfoFromFile()".
    -NEW: Added functionality to update individual UObjects now. Any class values can be filled just like a struct array. All wildcard struct inputs now accept struct arrays or UObject arrays.
    -NEW: A new module has been added, easyCSV. This is a faster implementation of the CSV parsing that is less prone to error and also supports special characters inside of quotes preceded by "". For instance, you can quote inside quotes with " or create a new line with \n. This module will appear on the marketplace as a separate plugin, but will be included for all users of Runtime DataTable in full.
    -NEW: Export functionality! Export does not support Google Sheets in this plugin, but it is now possible to export structs and object data to a new local spreadsheet! Node is called "ExportArrayToString". This CSV data as a string can then be saved to a local file or fed directly into a new data table. In C++, use GenerateCSV_FromArray().
    -NEW: Added New Parameter to UpdateObjectOrStructArrayFromPublicGoogleSheetsOrLocalCSV() and UpdateArrayFromGoogleSheetsCSV_Internal() named "MatchStructMemberNames" that when true will attempt to match column names in your CSV with variables inside of your struct. This makes it so you don't have to have all variables in your struct represented sequentially in your CSV file. Note, name matching is slower than sequential updates so when working with very large data sets updates could take sometime longer to complete. In most cases however you shouldn't notice any speed difference. This parameter has no effect when using an array of objects because objects will always use name matching. Names must match exactly in terms of characters used, but capitalization does not matter. When false, header names in the CSV don't need to match, just the order of the variables.
    -NEW: Some blueprint nodes now have less-often-used pins hidden behind a folding arrow. If you need them, just unfold the node.
    -BUGFIX: Nodes that do support Google Sheets are now marked as 'UnsafeDuringActorConstruction' so they will not be available in the Construction Script. If you call a Custom Event from the Construction Script that calls a node that is unsafe, the plugin will print an error to the screen in development builds and in editor.
    -BUGFIX: Made all nodes read an empty "OwningObject" pin as "Self" in blueprints. It no longer needs to be connected if the blueprint you're calling from owns the struct array or if the array is full of objects.
    -INFO: "Struct Array To Fill" pin on the UpdateObjectOrStructArrayFromPublicGoogleSheetsOrLocalCSV() node has been renamed as "Array To Update" to be clearer as it can accommodate struct arrays or object arrays now. You may need to update your node connections in blueprint.
    -INFO: In C++, the main method is now called UpdateArrayFromGoogleSheetsCSV_Internal(). The old method has been deprecated, so be sure to update your code. The new C++ method also has reordered parameters to allow for default parameters.
    -INFO: The following functions have been deprecated in the the DynamicDataTable module and have been moved to the included easyCSV module. Please update your references for the next release as they will be removed. The new versions of these functions require an FCSV_Info input which is a public variable saved to the Runtime DataTable Actor or can be generated using "MakeCSV_InfoFromString()" or "MakeCSV_InfoFromFile()":
    • SaveStringToFile
    • LoadStringFromFile
    • GetColumnAsStringArray
    • GetRowValueAsString
    • GetMapKeys
    • GetMapKeyIndex

    Leave a comment:


  • replied
    Oct 14 2020

    Hey everyone, couple of updates!

    First, there was an issue with my 4.24 fix that was supposed to fix support for accented characters. This issue has been fixed now and is live. Fixing my fix

    Second, on the 4.25+ version:

    File Update and Price Increase incoming!

    Coming in the next couple of weeks:
    1. Update Object variables directly as an alternative to creating structs (though structs are still an option)
    2. Variable order no longer matters, now matching by header name (or do it the old way if you prefer)
    3. Local CSV Export from structs or objects - you can even choose which variables get exported or skipped. Thank you everyone for your requests!
    4. Inclusion of easyCSV - this update includes a massive refactor of CSV parsing code which is much faster and reusable. If using a local CSV, it's also construction script safe. easyCSV is now a separate plugin embedded into Runtime DataTable.
    5. $15 USD will be the new price, so get it now to lock in the current price!
    Q&A:

    Why are you raising the price?


    Getting export working was not trivial. It involved synching up import and export functions to read and write data the exact same way depending on the data type itself and also be compatible with the way the engine exports data as string, so it involved a lot of trial and error and testing. Because of the amount of time it took, I thought it would be reasonable to ask for a little more than before rather than making it a separate plugin. This way all existing owners get the update for free. Thank you for being patient!

    In addition to export functionality, I'm polishing up a new feature that will affect variables on objects directly. No need to create structs anymore, though structs will of course still work. I was getting bothered by the cumbersome nature of having to add more members to my struct every time I wanted to link it to my Google Sheet. My preferred workflow is just to right-click a pin and promote it to variable, not go out of my way to update a struct and break it wherever I need it. You can still update structs the same way, but now you can update object variables directly too.

    Of course, this required a new method of populating variables from the sheet. Structs can be filled sequentially - the sequence of members in a class is not so clear-cut. A sequence exists, but is not easily apparent to the end user. So I decided instead to match the header names with the variable names. This is the only way to update object variables directly, so be sure to make your headers and variable names match if you want to go this route. If using a struct, you have the option of a sequential or name-matching update. If you don't want to have to update anything, just use sequential (default).

    Finally, I broke out the CSV parsing functionality into its own plugin. It's intended as a lower-cost alternative to RDT, for people dealing only in strings or those that don't mind converting the data on their own. It has no Google Sheets functionality. easyCSV, as I call it, will be available on the marketplace soon, but it is included with RDT for free in full. It is a faster parser that also supports new lines, commas and quotes inside quotes. It's just a much more robust solution which was necessary to develop to accommodate local export.

    I hope this added value feels worth it to potential buyers!

    Local Export? Does that mean there won't be Google Sheets export?

    That's correct, export is local-only. Google Sheets allows anonymous users to read public sheets, but you can't write anonymously to sheets at all through the API even if the sheet is public - you must authorize yourself. Thus, I decided not to include that functionality in this release. Reading public Google Sheets will work the same as it always has. No functionality has been removed from this release, only added.

    Will Google Sheets export come later?

    I am working on OAuth support for RDT, but it will come as a separate plugin. After my exploration of the idea, I found that the effort required to implement it would be pretty hefty. Given that knowledge, I decided that hiking the price up even further for this feature that most wouldn't use would be beneficial to the largest group, so when I pursue private sheets and exporting to sheets it will be for a separate product that makes OAuth its primary focus.

    Will these updates come to 4.24 or earlier?

    No, the only reason being that 4.25 introduced a major structural change in how properties are accessed which required a major refactor of the code. Retrofitting this into the 4.24 implementation would be a huge undertaking and I'm not sure all functionality would even work in 4.24, again because of the way properties are accessed. This new update will only be for 4.25 and newer.

    Will my workflow change?

    Yes and no. There will be some structural changes but won't affect everyone, and those who are affected will have until the next release to update their code. The full list of deprecated functions will be in the changelog when this update releases and the documentation will be updated to reflect the new changes as well. If all you do is fill structs with the basic node, you don't need to worry about code changes. This will only affect those using the RuntimeDataTableActor - you'll need to get a new struct inside of the Actor to get information like keys and headers or access cell data.
    Last edited by Jared Therriault; 10-14-2020, 10:57 PM.

    Leave a comment:


  • replied
    Originally posted by IgnacioFarias View Post
    Hi! Im developing for Android and Im not sure what I should use as the Save/Load Path. Im also using this plugin to download images from urls and save them, would I use this same path for that?

    Thanks!
    Hello IgnacioFarias,

    Save Path and Load Path are optional provided you're using an online Google Sheet. If you provide a path, the plugin will save the Google Sheet in CSV format. You can then load the CSV from the path in the event your game is offline or you don't provide a Google Sheets URL.

    If you want to provide a path, then the easiest method on mobile is to use relative paths through blueprint. Searching for "Paths" in blueprint nodes, you can build something like this:

    Click image for larger version  Name:	paths.PNG Views:	0 Size:	244.4 KB ID:	1802432

    UE4 will return the appropriate path based on the operating system this way, then the plugin will construct any missing folders as needed.

    As for your second question, this plugin won't download images directly. I assume that you're storing image URLs in a spreadsheet and downloading them separately, correct? If that's the case then you can save them anywhere that makes sense to you. The plugin doesn't have anything to do with that But as a general tip, I'd still recommend using the Paths blueprint nodes just for ease of use.
    Last edited by Jared Therriault; 08-18-2020, 01:23 PM.

    Leave a comment:


  • replied
    Hi! Im developing for Android and Im not sure what I should use as the Save/Load Path. Im also using this plugin to download images from urls and save them, would I use this same path for that?

    Thanks!

    Leave a comment:


  • replied
    Originally posted by Jared Therriault View Post

    Taking this 1-to-1 so I can ask about project-specific information.
    For those reading, we worked this out. For some reason the marketplace gave this an outdated version of the plugin - if you're on 4.24 and experience issues with non-latin writing systems when using this plugin let me know.

    Leave a comment:


  • replied
    Originally posted by Crusader_91 View Post
    Hello Jared,

    This is a duplicate of the letter I sent you personally.

    We were talking about an issue about encoding cyrillic symbols in the downloaded information. You asked me to send the payload string to you. Here it is (partially):


    ========================================================================
    Code:
    LogBlueprintUserMessages: [BP_ItemManager_2] ---,Item Name: Identified,Item Class,Item Type,Item Mesh In World,Item Materials,Item UI Image,Description,Usable,Equipable,Stackable,Max Stack Count,Item Price,OneItemMass,PivotPointShiftInItemInspector,EventsToActivateOnPickup
    00000000,"(RU=""Масло навыка: Перехват"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_Intercept.BP_Equip_Oil_Skill_General_Intercept_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/MI_ItemsUI_Oil_Skill_Gener
    al_Intercept.MI_ItemsUI_Oil_Skill_General_Intercept,"(RU=""Капсула с Маслом, дающая персонажу навык: Перехват"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
    00000001,"(RU=""Масло навыка: Дезориентирующий приказ"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_DisorientOrder.BP_Equip_Oil_Skill_General_DisorientOrder_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/M
    I_ItemsUI_Oil_Skill_General_Disorient.MI_ItemsUI_Oil_Skill_General_Disorient,"(RU=""Капсула с Маслом, дающая персонажу навык: Дезориентирующий приказ"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
    ========================================================================

    As I can see, the payload string is correct, so the garbage data is definitely not coming from Google. Each segment of the Russian-language text (after the labels "RU") is reproduced correctly.

    I also attach a link to a copy of this table, so you can make sure the information is correct:

    https://docs.google.com/spreadsheets...it?usp=sharing

    Thanks for your work again!
    Taking this 1-to-1 so I can ask about project-specific information.

    Leave a comment:


  • replied
    Originally posted by joesys View Post
    Hi Jared,

    Thank you for the wonderful plugin.

    I notice the following code in ReadCSVRow()

    Code:
    //Ensure we won't end up with an index under 0 (in case of an empty string)
    int32 FieldIndex = (fields[CSV_FieldIndex].Len() - 1 < 0 ? 0 : fields[CSV_FieldIndex].Len() - 1); //todo: keep an eye on this
    
    if (fields[CSV_FieldIndex].Len() > 1 && fields[CSV_FieldIndex][FieldIndex] == '\\') //if we find a wrapper character and the last character was supposed to escape it then it was obviously included by the user
    {
    fields[CSV_FieldIndex][FieldIndex] = WrapperCharacter;
    StateIndex = 3; //Quote within user string/text
    }
    
    StateIndex = 2; //Wrapped wrapper
    I believe it should be:

    Code:
    ...
    
    StateIndex = 3; //Quote within user string/text
    }
    else
    {
    StateIndex = 2; //Wrapped wrapper
    }
    Otherwise, StateIndex will always be 2.
    You're right joesys! I actually saw this oversight a few weeks ago and fixed it localy but I haven't pushed the fix. I will be pushing a fix soon (in the next month). This should only affect users who have double-nested quotes in their text and heavy use of "\". For most users this won't be an issue, but a bug is a bug and I'll have the fix out soon! Thank you!

    Leave a comment:


  • replied
    Hello Jared,

    This is a duplicate of the letter I sent you personally.

    We were talking about an issue about encoding cyrillic symbols in the downloaded information. You asked me to send the payload string to you. Here it is (partially):


    ========================================================================
    Code:
    LogBlueprintUserMessages: [BP_ItemManager_2] ---,Item Name: Identified,Item Class,Item Type,Item Mesh In World,Item Materials,Item UI Image,Description,Usable,Equipable,Stackable,Max Stack Count,Item Price,OneItemMass,PivotPointShiftInItemInspector,EventsToActivateOnPickup
    00000000,"(RU=""Масло навыка: Перехват"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_Intercept.BP_Equip_Oil_Skill_General_Intercept_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/MI_ItemsUI_Oil_Skill_Gener
    al_Intercept.MI_ItemsUI_Oil_Skill_General_Intercept,"(RU=""Капсула с Маслом, дающая персонажу навык: Перехват"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
    00000001,"(RU=""Масло навыка: Дезориентирующий приказ"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_DisorientOrder.BP_Equip_Oil_Skill_General_DisorientOrder_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/M
    I_ItemsUI_Oil_Skill_General_Disorient.MI_ItemsUI_Oil_Skill_General_Disorient,"(RU=""Капсула с Маслом, дающая персонажу навык: Дезориентирующий приказ"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
    ========================================================================

    As I can see, the payload string is correct, so the garbage data is definitely not coming from Google. Each segment of the Russian-language text (after the labels "RU") is reproduced correctly.

    I also attach a link to a copy of this table, so you can make sure the information is correct:

    https://docs.google.com/spreadsheets...it?usp=sharing

    Thanks for your work again!
    Last edited by Crusader_91; 07-17-2020, 02:16 PM.

    Leave a comment:


  • replied
    Originally posted by Crusader_91 View Post
    Hello, Jared!

    I have encountered an issue working with your plugin.

    It doesn't encode cyrillic symbols correctly after import from Google spreadsheets. For example, russian word "Приказ" was encoded to "?@8:07". I think, that's because of default UTF-16 encoding. Is it possible to use UTF-8 encoding during import?
    Hi Crusader_91, maybe you can help me figure out when this is happening. After 'Call On Complete' is called, save the payload and send that to me please!

    I'm currently using wchar_t to parse the payload into structs and it's my understanding that should support Cyrillic, so I would like to know if the garbage data is coming from Google or if the script converts it to garbage. Send some samples when you can! Thank you.

    Leave a comment:


  • replied
    Hello, Jared!

    I have encountered an issue working with your plugin.

    It doesn't encode cyrillic symbols correctly after import from Google spreadsheets. For example, russian word "Приказ" was encoded to "?@8:07". I think, that's because of default UTF-16 encoding. Is it possible to use UTF-8 encoding during import?

    Leave a comment:

Working...
X