Announcement

Collapse
No announcement yet.

Runtime DataTable - Import text CSV or Google Sheet at runtime and fill an array of structs!

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #31
    Originally posted by danibb View Post
    Would it be possible to export a data structure to excell in runtime?
    Hi danibb - not with this plugin at this time, but this is a request I see a lot. It is something I will consider for the future.
    iTween For UE4
    Procedural pathed animation for actors, components & UMG - $0
    Runtime DataTable
    DataTables dynamically loaded from text or Google Sheets while your game is running!
    easyCSV
    Fast, cheap, and accurate CSV parsing at runtime!

    Comment


      #32
      Originally posted by danibb View Post
      Would it be possible to export a data structure to excell in runtime?
      I second this request! I have some complex structs that would be great to export and be able to manipulate. I'm sure I read somewhere that this supports structs within structs, correct? It would be nice to be able to export just to make sure that a complex CSV like that is setup correctly for import.

      Comment


        #33
        Originally posted by AaronMTodd View Post

        I second this request! I have some complex structs that would be great to export and be able to manipulate. I'm sure I read somewhere that this supports structs within structs, correct? It would be nice to be able to export just to make sure that a complex CSV like that is setup correctly for import.
        That's a fair use case, thank you for the feedback! It is true, the plugin supports structs within structs. For now, I'd recommend creating a normal DataTable in the engine with a nested struct then exporting it to CSV to get a feel for the layout, then create new rows in Google sheets based on that.

        The heat is rising on this feature request, but I'll be honest: This is something I've looked into and the amount of work for implementation wouldn't be trivial. So if I do make this feature happen it will be part of a separate plugin. Reasons being:
        • I don't want to raise the price of RDT as it is
        • People who don't need the export feature might feel burned having to pay more when it didn't use to be like that
        • The time investment for implementation, testing and support will far surpass the amount of money I make from the plugin to begin with
        So it would be a separate buy-in. I'm not sure how customers would react to that. How do you feel?
        iTween For UE4
        Procedural pathed animation for actors, components & UMG - $0
        Runtime DataTable
        DataTables dynamically loaded from text or Google Sheets while your game is running!
        easyCSV
        Fast, cheap, and accurate CSV parsing at runtime!

        Comment


          #34
          Hi Jared,

          Thank you for the wonderful plugin.

          I notice the following code in ReadCSVRow()

          Code:
          //Ensure we won't end up with an index under 0 (in case of an empty string)
          int32 FieldIndex = (fields[CSV_FieldIndex].Len() - 1 < 0 ? 0 : fields[CSV_FieldIndex].Len() - 1); //todo: keep an eye on this
          
          if (fields[CSV_FieldIndex].Len() > 1 && fields[CSV_FieldIndex][FieldIndex] == '\\') //if we find a wrapper character and the last character was supposed to escape it then it was obviously included by the user
          {
              fields[CSV_FieldIndex][FieldIndex] = WrapperCharacter;
              StateIndex = 3; //Quote within user string/text
          }
          
          StateIndex = 2; //Wrapped wrapper
          I believe it should be:

          Code:
          ...
          
              StateIndex = 3; //Quote within user string/text
          }
          else 
          {
              StateIndex = 2; //Wrapped wrapper
          }
          Otherwise, StateIndex will always be 2.

          Comment


            #35
            Hello, Jared!

            I have encountered an issue working with your plugin.

            It doesn't encode cyrillic symbols correctly after import from Google spreadsheets. For example, russian word "Приказ" was encoded to "?@8:07". I think, that's because of default UTF-16 encoding. Is it possible to use UTF-8 encoding during import?

            Comment


              #36
              Originally posted by Crusader_91 View Post
              Hello, Jared!

              I have encountered an issue working with your plugin.

              It doesn't encode cyrillic symbols correctly after import from Google spreadsheets. For example, russian word "Приказ" was encoded to "?@8:07". I think, that's because of default UTF-16 encoding. Is it possible to use UTF-8 encoding during import?
              Hi Crusader_91, maybe you can help me figure out when this is happening. After 'Call On Complete' is called, save the payload and send that to me please!

              I'm currently using wchar_t to parse the payload into structs and it's my understanding that should support Cyrillic, so I would like to know if the garbage data is coming from Google or if the script converts it to garbage. Send some samples when you can! Thank you.
              iTween For UE4
              Procedural pathed animation for actors, components & UMG - $0
              Runtime DataTable
              DataTables dynamically loaded from text or Google Sheets while your game is running!
              easyCSV
              Fast, cheap, and accurate CSV parsing at runtime!

              Comment


                #37
                Hello Jared,

                This is a duplicate of the letter I sent you personally.

                We were talking about an issue about encoding cyrillic symbols in the downloaded information. You asked me to send the payload string to you. Here it is (partially):


                ========================================================================
                Code:
                LogBlueprintUserMessages: [BP_ItemManager_2] ---,Item Name: Identified,Item Class,Item Type,Item Mesh In World,Item Materials,Item UI Image,Description,Usable,Equipable,Stackable,Max Stack Count,Item Price,OneItemMass,PivotPointShiftInItemInspector,EventsToActivateOnPickup
                00000000,"(RU=""Масло навыка: Перехват"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_Intercept.BP_Equip_Oil_Skill_General_Intercept_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/MI_ItemsUI_Oil_Skill_Gener
                al_Intercept.MI_ItemsUI_Oil_Skill_General_Intercept,"(RU=""Капсула с Маслом, дающая персонажу навык: Перехват"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
                00000001,"(RU=""Масло навыка: Дезориентирующий приказ"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_DisorientOrder.BP_Equip_Oil_Skill_General_DisorientOrder_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/M
                I_ItemsUI_Oil_Skill_General_Disorient.MI_ItemsUI_Oil_Skill_General_Disorient,"(RU=""Капсула с Маслом, дающая персонажу навык: Дезориентирующий приказ"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
                ========================================================================

                As I can see, the payload string is correct, so the garbage data is definitely not coming from Google. Each segment of the Russian-language text (after the labels "RU") is reproduced correctly.

                I also attach a link to a copy of this table, so you can make sure the information is correct:

                https://docs.google.com/spreadsheets...it?usp=sharing

                Thanks for your work again!
                Last edited by Crusader_91; 07-17-2020, 02:16 PM.

                Comment


                  #38
                  Originally posted by joesys View Post
                  Hi Jared,

                  Thank you for the wonderful plugin.

                  I notice the following code in ReadCSVRow()

                  Code:
                  //Ensure we won't end up with an index under 0 (in case of an empty string)
                  int32 FieldIndex = (fields[CSV_FieldIndex].Len() - 1 < 0 ? 0 : fields[CSV_FieldIndex].Len() - 1); //todo: keep an eye on this
                  
                  if (fields[CSV_FieldIndex].Len() > 1 && fields[CSV_FieldIndex][FieldIndex] == '\\') //if we find a wrapper character and the last character was supposed to escape it then it was obviously included by the user
                  {
                  fields[CSV_FieldIndex][FieldIndex] = WrapperCharacter;
                  StateIndex = 3; //Quote within user string/text
                  }
                  
                  StateIndex = 2; //Wrapped wrapper
                  I believe it should be:

                  Code:
                  ...
                  
                  StateIndex = 3; //Quote within user string/text
                  }
                  else
                  {
                  StateIndex = 2; //Wrapped wrapper
                  }
                  Otherwise, StateIndex will always be 2.
                  You're right joesys! I actually saw this oversight a few weeks ago and fixed it localy but I haven't pushed the fix. I will be pushing a fix soon (in the next month). This should only affect users who have double-nested quotes in their text and heavy use of "\". For most users this won't be an issue, but a bug is a bug and I'll have the fix out soon! Thank you!
                  iTween For UE4
                  Procedural pathed animation for actors, components & UMG - $0
                  Runtime DataTable
                  DataTables dynamically loaded from text or Google Sheets while your game is running!
                  easyCSV
                  Fast, cheap, and accurate CSV parsing at runtime!

                  Comment


                    #39
                    Originally posted by Crusader_91 View Post
                    Hello Jared,

                    This is a duplicate of the letter I sent you personally.

                    We were talking about an issue about encoding cyrillic symbols in the downloaded information. You asked me to send the payload string to you. Here it is (partially):


                    ========================================================================
                    Code:
                    LogBlueprintUserMessages: [BP_ItemManager_2] ---,Item Name: Identified,Item Class,Item Type,Item Mesh In World,Item Materials,Item UI Image,Description,Usable,Equipable,Stackable,Max Stack Count,Item Price,OneItemMass,PivotPointShiftInItemInspector,EventsToActivateOnPickup
                    00000000,"(RU=""Масло навыка: Перехват"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_Intercept.BP_Equip_Oil_Skill_General_Intercept_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/MI_ItemsUI_Oil_Skill_Gener
                    al_Intercept.MI_ItemsUI_Oil_Skill_General_Intercept,"(RU=""Капсула с Маслом, дающая персонажу навык: Перехват"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
                    00000001,"(RU=""Масло навыка: Дезориентирующий приказ"",EN=""None"")",/Game/Blueprints/World/Items/Oil_Items/Skills/BP_Equip_Oil_Skill_General_DisorientOrder.BP_Equip_Oil_Skill_General_DisorientOrder_C,Oil_Skill,/Game/Prototyping/Meshes/Items/Potionbottle_001.Potionbottle_001,(None),/Game/VisualContent/UI/GraphicsAndSprites/EquipmentIconsAndItems/Items/Oil/M
                    I_ItemsUI_Oil_Skill_General_Disorient.MI_ItemsUI_Oil_Skill_General_Disorient,"(RU=""Капсула с Маслом, дающая персонажу навык: Дезориентирующий приказ"",EN=""English Text"")",FALSE,TRUE,FALSE,0,0,0.500000,"(X=0.000000,Y=0.000000,Z=0.000000)",None
                    ========================================================================

                    As I can see, the payload string is correct, so the garbage data is definitely not coming from Google. Each segment of the Russian-language text (after the labels "RU") is reproduced correctly.

                    I also attach a link to a copy of this table, so you can make sure the information is correct:

                    https://docs.google.com/spreadsheets...it?usp=sharing

                    Thanks for your work again!
                    Taking this 1-to-1 so I can ask about project-specific information.
                    iTween For UE4
                    Procedural pathed animation for actors, components & UMG - $0
                    Runtime DataTable
                    DataTables dynamically loaded from text or Google Sheets while your game is running!
                    easyCSV
                    Fast, cheap, and accurate CSV parsing at runtime!

                    Comment


                      #40
                      Originally posted by Jared Therriault View Post

                      Taking this 1-to-1 so I can ask about project-specific information.
                      For those reading, we worked this out. For some reason the marketplace gave this an outdated version of the plugin - if you're on 4.24 and experience issues with non-latin writing systems when using this plugin let me know.
                      iTween For UE4
                      Procedural pathed animation for actors, components & UMG - $0
                      Runtime DataTable
                      DataTables dynamically loaded from text or Google Sheets while your game is running!
                      easyCSV
                      Fast, cheap, and accurate CSV parsing at runtime!

                      Comment


                        #41
                        Hi! Im developing for Android and Im not sure what I should use as the Save/Load Path. Im also using this plugin to download images from urls and save them, would I use this same path for that?

                        Thanks!

                        Comment


                          #42
                          Originally posted by IgnacioFarias View Post
                          Hi! Im developing for Android and Im not sure what I should use as the Save/Load Path. Im also using this plugin to download images from urls and save them, would I use this same path for that?

                          Thanks!
                          Hello IgnacioFarias,

                          Save Path and Load Path are optional provided you're using an online Google Sheet. If you provide a path, the plugin will save the Google Sheet in CSV format. You can then load the CSV from the path in the event your game is offline or you don't provide a Google Sheets URL.

                          If you want to provide a path, then the easiest method on mobile is to use relative paths through blueprint. Searching for "Paths" in blueprint nodes, you can build something like this:

                          Click image for larger version  Name:	paths.PNG Views:	0 Size:	244.4 KB ID:	1802432

                          UE4 will return the appropriate path based on the operating system this way, then the plugin will construct any missing folders as needed.

                          As for your second question, this plugin won't download images directly. I assume that you're storing image URLs in a spreadsheet and downloading them separately, correct? If that's the case then you can save them anywhere that makes sense to you. The plugin doesn't have anything to do with that But as a general tip, I'd still recommend using the Paths blueprint nodes just for ease of use.
                          Last edited by Jared Therriault; 08-18-2020, 01:23 PM.
                          iTween For UE4
                          Procedural pathed animation for actors, components & UMG - $0
                          Runtime DataTable
                          DataTables dynamically loaded from text or Google Sheets while your game is running!
                          easyCSV
                          Fast, cheap, and accurate CSV parsing at runtime!

                          Comment


                            #43
                            Oct 14 2020

                            Hey everyone, couple of updates!

                            First, there was an issue with my 4.24 fix that was supposed to fix support for accented characters. This issue has been fixed now and is live. Fixing my fix

                            Second, on the 4.25+ version:

                            File Update and Price Increase incoming!

                            Coming in the next couple of weeks:
                            1. Update Object variables directly as an alternative to creating structs (though structs are still an option)
                            2. Variable order no longer matters, now matching by header name (or do it the old way if you prefer)
                            3. Local CSV Export from structs or objects - you can even choose which variables get exported or skipped. Thank you everyone for your requests!
                            4. Inclusion of easyCSV - this update includes a massive refactor of CSV parsing code which is much faster and reusable. If using a local CSV, it's also construction script safe. easyCSV is now a separate plugin embedded into Runtime DataTable.
                            5. $15 USD will be the new price, so get it now to lock in the current price!
                            Q&A:

                            Why are you raising the price?


                            Getting export working was not trivial. It involved synching up import and export functions to read and write data the exact same way depending on the data type itself and also be compatible with the way the engine exports data as string, so it involved a lot of trial and error and testing. Because of the amount of time it took, I thought it would be reasonable to ask for a little more than before rather than making it a separate plugin. This way all existing owners get the update for free. Thank you for being patient!

                            In addition to export functionality, I'm polishing up a new feature that will affect variables on objects directly. No need to create structs anymore, though structs will of course still work. I was getting bothered by the cumbersome nature of having to add more members to my struct every time I wanted to link it to my Google Sheet. My preferred workflow is just to right-click a pin and promote it to variable, not go out of my way to update a struct and break it wherever I need it. You can still update structs the same way, but now you can update object variables directly too.

                            Of course, this required a new method of populating variables from the sheet. Structs can be filled sequentially - the sequence of members in a class is not so clear-cut. A sequence exists, but is not easily apparent to the end user. So I decided instead to match the header names with the variable names. This is the only way to update object variables directly, so be sure to make your headers and variable names match if you want to go this route. If using a struct, you have the option of a sequential or name-matching update. If you don't want to have to update anything, just use sequential (default).

                            Finally, I broke out the CSV parsing functionality into its own plugin. It's intended as a lower-cost alternative to RDT, for people dealing only in strings or those that don't mind converting the data on their own. It has no Google Sheets functionality. easyCSV, as I call it, will be available on the marketplace soon, but it is included with RDT for free in full. It is a faster parser that also supports new lines, commas and quotes inside quotes. It's just a much more robust solution which was necessary to develop to accommodate local export.

                            I hope this added value feels worth it to potential buyers!

                            Local Export? Does that mean there won't be Google Sheets export?

                            That's correct, export is local-only. Google Sheets allows anonymous users to read public sheets, but you can't write anonymously to sheets at all through the API even if the sheet is public - you must authorize yourself. Thus, I decided not to include that functionality in this release. Reading public Google Sheets will work the same as it always has. No functionality has been removed from this release, only added.

                            Will Google Sheets export come later?

                            I am working on OAuth support for RDT, but it will come as a separate plugin. After my exploration of the idea, I found that the effort required to implement it would be pretty hefty. Given that knowledge, I decided that hiking the price up even further for this feature that most wouldn't use would be beneficial to the largest group, so when I pursue private sheets and exporting to sheets it will be for a separate product that makes OAuth its primary focus.

                            Will these updates come to 4.24 or earlier?

                            No, the only reason being that 4.25 introduced a major structural change in how properties are accessed which required a major refactor of the code. Retrofitting this into the 4.24 implementation would be a huge undertaking and I'm not sure all functionality would even work in 4.24, again because of the way properties are accessed. This new update will only be for 4.25 and newer.

                            Will my workflow change?

                            Yes and no. There will be some structural changes but won't affect everyone, and those who are affected will have until the next release to update their code. The full list of deprecated functions will be in the changelog when this update releases and the documentation will be updated to reflect the new changes as well. If all you do is fill structs with the basic node, you don't need to worry about code changes. This will only affect those using the RuntimeDataTableActor - you'll need to get a new struct inside of the Actor to get information like keys and headers or access cell data.
                            Last edited by Jared Therriault; 10-14-2020, 10:57 PM.
                            iTween For UE4
                            Procedural pathed animation for actors, components & UMG - $0
                            Runtime DataTable
                            DataTables dynamically loaded from text or Google Sheets while your game is running!
                            easyCSV
                            Fast, cheap, and accurate CSV parsing at runtime!

                            Comment


                              #44
                              Hi everyone! The Big Island update has been pushed to the marketplace, now we're just waiting on Epic approval! Keep in mind that the price will increase to $15 USD when the update is approved, so this is your last chance to pick it up at the current price!

                              Highlights:
                              • Update Object variables directly as an alternative to creating structs
                              • Variable order no longer matters, now matching by header name (or do it the old way if you prefer)
                              • Local CSV Export from structs or objects - you can even choose which variables get exported or skipped
                              • Inclusion of easyCSV - this update includes a massive refactor of CSV parsing code which is much faster and reusable. If using a local CSV, it's also construction script safe.

                              Full Changelog:
                              ("Big Island" Update, Nov 1 2020):
                              -NEW: Added Construction Safe version of the node that does not support Google Sheets but can be called from the construction script. It's called "UpdateArrayFromCSV_Info()" and updates the array input values on the blocking game thread from an FCSV_Info struct which can be generated using "MakeCSV_InfoFromString()" or "MakeCSV_InfoFromFile()".
                              -NEW: Added functionality to update individual UObjects now. Any class values can be filled just like a struct array. All wildcard struct inputs now accept struct arrays or UObject arrays.
                              -NEW: A new module has been added, easyCSV. This is a faster implementation of the CSV parsing that is less prone to error and also supports special characters inside of quotes preceded by "". For instance, you can quote inside quotes with " or create a new line with \n. This module will appear on the marketplace as a separate plugin, but will be included for all users of Runtime DataTable in full.
                              -NEW: Export functionality! Export does not support Google Sheets in this plugin, but it is now possible to export structs and object data to a new local spreadsheet! Node is called "ExportArrayToString". This CSV data as a string can then be saved to a local file or fed directly into a new data table. In C++, use GenerateCSV_FromArray().
                              -NEW: Added New Parameter to UpdateObjectOrStructArrayFromPublicGoogleSheetsOrLocalCSV() and UpdateArrayFromGoogleSheetsCSV_Internal() named "MatchStructMemberNames" that when true will attempt to match column names in your CSV with variables inside of your struct. This makes it so you don't have to have all variables in your struct represented sequentially in your CSV file. Note, name matching is slower than sequential updates so when working with very large data sets updates could take sometime longer to complete. In most cases however you shouldn't notice any speed difference. This parameter has no effect when using an array of objects because objects will always use name matching. Names must match exactly in terms of characters used, but capitalization does not matter. When false, header names in the CSV don't need to match, just the order of the variables.
                              -NEW: Some blueprint nodes now have less-often-used pins hidden behind a folding arrow. If you need them, just unfold the node.
                              -BUGFIX: Nodes that do support Google Sheets are now marked as 'UnsafeDuringActorConstruction' so they will not be available in the Construction Script. If you call a Custom Event from the Construction Script that calls a node that is unsafe, the plugin will print an error to the screen in development builds and in editor.
                              -BUGFIX: Made all nodes read an empty "OwningObject" pin as "Self" in blueprints. It no longer needs to be connected if the blueprint you're calling from owns the struct array or if the array is full of objects.
                              -INFO: "Struct Array To Fill" pin on the UpdateObjectOrStructArrayFromPublicGoogleSheetsOrLocalCSV() node has been renamed as "Array To Update" to be clearer as it can accommodate struct arrays or object arrays now. You may need to update your node connections in blueprint.
                              -INFO: In C++, the main method is now called UpdateArrayFromGoogleSheetsCSV_Internal(). The old method has been deprecated, so be sure to update your code. The new C++ method also has reordered parameters to allow for default parameters.
                              -INFO: The following functions have been deprecated in the the DynamicDataTable module and have been moved to the included easyCSV module. Please update your references for the next release as they will be removed. The new versions of these functions require an FCSV_Info input which is a public variable saved to the Runtime DataTable Actor or can be generated using "MakeCSV_InfoFromString()" or "MakeCSV_InfoFromFile()":
                              • SaveStringToFile
                              • LoadStringFromFile
                              • GetColumnAsStringArray
                              • GetRowValueAsString
                              • GetMapKeys
                              • GetMapKeyIndex
                              iTween For UE4
                              Procedural pathed animation for actors, components & UMG - $0
                              Runtime DataTable
                              DataTables dynamically loaded from text or Google Sheets while your game is running!
                              easyCSV
                              Fast, cheap, and accurate CSV parsing at runtime!

                              Comment

                              Working...
                              X