Runtime DataTable - Import/Export text CSV or Google Sheet at runtime and auto-update structs and UObjects!

Ok, so this should work just fine given you don’t need any other data in the CSV.

Pull out of the Test Struct Array variable pin and call “Get.” Pull out of the same pin again and call “Last Index” and plug it into the Index pin on the Get node. This will always get the final row of the struct.

Finally, pull out of the struct output pin on the “Get” node and choose “Break {Struct Name}.” Here you should be able to access “value” and do what you need to do.

Okay that works well, thanks. This is how it looks now.

The only problem I have is that this only works when I close my csv recording app and not on runtime. The print string stays empty… How I can fix that? Like I said the csv is saved every second.

This part is an OS limitation - if your program locks the file whilst writing to its IO stream then Unreal Engine can’t open it to read it.

If you can, edit the recording application to not lock the file, open the IO stream only when a write is needed and close it when it’s not, or write to something like Google Sheets.

If you can’t edit the recording application or stream/write the data in some other way, you may need to write a secondary application with elevated permissions that makes a copy of the file each second then use that copy instead.

It works now. I changed some parts inside the blueprint but I’m getting this error for some reason…

Another question I have is how I can make Unreal choose the csv file automatically? The recording app saves the files in order per session…

  1. Accessed None means there is no last index (I.E. the array is empty). Before you set LastBPMValue, throw a branch node in that checks if TestStructArray has a Length > 0.

  2. It looks like it saves the CSV files odinally (1, 2, 3… 12). You can either keep track of the last number CSV you referenced with an integer variable and increase it by one each time you are ready to pull the data then convert the integer variable to string. Just pull out of the pin and type “to string”. If that doesn’t work for you, you could try out Rama’s BP plugin which has a “Get Files” function. Get all the files in the folder and figure out which one you want to read. If you only have the latest CSV files in that folder you could get the length of the Files array from that node, convert it to string and use that to get the file you need.

Rama’s Plugin is found here: (39) Rama's Extra Blueprint Nodes for You as a Plugin, No C++ Required!

Refercing the CSV to an integer works nice! Thank you

I added also the Branch node but still getting an error.

Okay I was wrong here… It doesn’t work at runtime… Only when I close the recording app.
I tried to change the csv also manually with VS code but nothing happend it keeps the value from the previous save. So you say this is a OS problem?
Is it also possible to create a copy of the file inside UE?

Correct, this is the recording app opening the I/O stream but not closing it until it finishes recording. The OS considers the file locked while the stream is open, so other applications can’t open the file until the stream is closed.

Unreal only requests read access when loading text from a file, so that means that the recording application is requesting exclusive access to the text file and that can only be solved by editing the code of the recording application unless it has an exposed option for non-exclusive write access. Most OSes (including Windows) allow you to copy the file without the exclusion even while the file is locked, but you’ll have to handle that outside of Unreal. Unreal can make copies but only by reading the original text file and writing it to a new one, which is impossible if the file is locked with read-exclusion.

Is your recording application something custom-made or is it a software solution from a third party? @deedox

Hi Jared, thank you now it’s more clear to me…
The recording app is custom made by another dev. I will ask him to join the discussion.

Hi Jared,

I am using a NodeJS script to write data to a CSV file each second. Yes, keeping the write stream open throughout since the publishing is happening each second. Can you suggest a way to write data in a “non-exclusive” way so that Unreal can read it simultaneously ?

I am using the “a+” file system flag which allows me to both read and append the file.

There’s an exclusive flag: “ax+” too. It throws errors when we try to open a file which doesn’t exist.

“a+” automatically creates the file using the supplied file name if it doesn’t exist

While the IO write process was on, I triggered another function (process) to read the content of that file simultaneously. Was able to do so without any errors:

const {readFile} = require("fs");
readFile("./##.csv", (_err, data) => {

This also worked:

const {createReadStream} = require("fs");
const strm = createReadStream(`${__dirname}/13.csv`, {start: 0});;
strm.on("data", (d) => {

Tested on both WIndows and macOS.

Is there a possibility that Unreal/plugin is requesting exclusive access instead?

@JaredTherriault Please let me know when you get a chance

Hello , I am trying to integrate my project from 4.26 to 4.27. In 4.26, I ran the plugin thanks to a variable named runtime data table actor. I can’t find this variable in 4.27. I guess remove it.I am sharing the problem in the screenshot. What can I put instead of runtime data table actor ?

1 Like

I don’t believe so. RDT leverages IFileManager::CreateFileReader with no flags. Looking at the code in the implementation, it doesn’t have any code I can sniff out that requires exclusivity or write access (which is something defined in the flags parameter).

Is there any way you can write to a new file on each loop or close the write stream between writes?

Hello emincelik, the actor that controlled these operations has been replaced with a UObject to allow for operations to complete at design time, not just runtime. But everything that you could do with the actor can now be done with just the CSV Info. From your screenshot, it looks like you’re just checking the validity of the actor then destroying it - is there anything else you’re using it for that you need some help with finding an alternative?

Creating a new copy of the original CSV each second also didn’t work. Perhaps the default (no flag) indicates exclusivity? The thing is that when a custom script is used to read the CSV simultaneously it works. If the recorded app was requesting exclusive access that should have failed too

Would you be able to send me the recording program with some mock data? It’s not possible to debug this without a repro at this point. I’ll need to step through the code but I don’t have a way to trigger this issue.

Hi Jared,

Great plugin - any chance you could update the code to remove the deprecation warnings? I know these are only deprecation warnings in prep for 5.1 and thus doesn’t necessarily need addressing immediately. However, it would be nice to stop these warnings appearing during output logs for code compiles + project packaging.

Many thanks,

Hello AgentMilkshake1,

I totally understand not enjoying all those deprecation warnings - I definitely don’t either! But they are important because 1) It stops new users from being able to access those functions and 2) It helps existing users update their code to the new functions.

Now the reason that you’re seeing it so much when packaging is because of the OneNode struct that was used to pass information around in the old style workflow. This whole workflow has been deprecated now but was the only way to get anything done previously so a LOT of users are still using this code. Therefore I can’t just remove it immediately. Many people would be upset that their code stopped working when they thought they had until 5.1 to fix it!

So I’m sorry but I’ve got to stick to my guns here. It sounds like you have some C++ skills though, so an easy fix in your situation would be:

  1. Move Runtime DataTable from {Engine}/Plugins/Marketplace to your project’s Plugins folder
  2. Go to line 134 in Plugins/RuntimeDataTable/Source/RuntimeDataTable/Public/RuntimeDataTable.h
  3. Remove the UE_DEPRECATED macro between “struct” and “FOneNodeInfoStruct”
  4. Compile

That should take care of all but two warnings. If you want to take care of the rest, you can just remove all deprecation warnings (or all deprecated functions if you’re so inclined).

1 Like