How to load big Data Tables faster?

Development Programming & Scripting
, ,
1

I have a good amount of Data Tables for my game that has Dialogues, face animations, body animations, subtitle text etc. Now every chapter and chapter’s segment calls it’s own DT Like Ch1 Seg1, Ch4 Seg5 etc.

I use my GameInstance to sort out DT as needed, and play dialogues as needed (dialogues along with animations, subtitles etc).

That’s how DTs are sorted

image
image1159×738 141 KB

Initially, I was using hard references then I saw the Size Map and it was around 27Gb for Disk and 6Gb for Memory. Then I switched to soft ref. Now it’s 3.5Gb disk and 1.5Gb memory. But due to that, I had to use a new node called AsyncLoadAsset then Cast to DT Class then set the Required Data Table in the variable.

This is the Set DT Fn, that fetches whichever DT is required according to Chapter/Segment. Converts soft Ref to hard ref

image
image1046×402 86.1 KB

Now it does work, but there are 2 issues .
First, it loads slowly . It’s not instantaneous like hard ref were.
Second, it still creates hitches as it loads (unreal shows on top right “preparing sound wave preparing animation asset” etc).
Now keeping in mind, I re-installed Windows a few days ago so maybe there’s no Asset Cache memory of all the DTs, hence this loading issue. But my question is, will this be a problem when I build the game?

This screenshot shows the Play Dialogues Fn that loads the DT and fetches data

image
image1158×617 75.3 KB

The game is intertwined to dialogues like let’s say a new Chapter is triggered. There’s a Dispatcher that is called inside the current level to do task accordingly (like lets say its Ch3 Seg1 and NPC A needs to go here B needs to do this etc).

The screenshot below shows external scenario for updating Ch/Seg. First node sets the Chapter/Segment and calls all dispatchers/TriggerBoxes and updates them about the same. Second node simply Plays the dialogues.

image
image984×742 139 KB

So like in the above screenshot, there are a good few moments when Dialogues are played instantly after setting new Ch/Seg. The problem is, it will cause issues bcz Level will run it’s code assuming dialogues are playing. But it won’t look proper as there will be a delay + hitching as game loads the DTs.
In some cases, dialogues are played right after triggering the next chapter node, whereas in other cases, there may be some delay in playing dialogues. The latter case is fine but the former is not, as there will be a disparity as DT is loaded and then dialogues are played, due to the delay of loading. Not to mention the hitching.

So now, again, will the final build have this issue ? That’s my main concern. I can continue my work knowing the final build won’t be affected.
And, is there a way to load certain assets (esp DTs) on the main menu as the game loads so that hitching is prevented?

2

there is no easy answer here,

the trick is to know what you need loaded and when, for instance know what dialog is needed on a level and load it then, or use a trigger volume so you can load the NPC dialog when a player is in a certain range.

I personally prefer data assets, so each NPC could have his own asset and knows exactly what to load.

of course you could just load everything at the start hidden behind a load screen.
and to answer your question, yes the final build will have these issues but also lower spec machines could be even worse

1 Like
3

I see your point but I guess I’m too deep tangled in this to now edit all my DT and make them data assets.
Can you guide me through the loading process? I can surely mark the DTs required by each level when player loads the game from save (depending on what chapter they were). Better to load it before than load it during gameplay.
Is there a node/command I need to put in for loading at the start?

Thanks.

4

before is easier, but requires you to store it all in memory
during gameplay is trickier but you can load/unload as required.

If you know what you need loaded, you’ll know how big it is so you could probably do it on level start. You could create some sort of ‘Loading’ actor or system that creates a loading widget and loads all the assets Async.

1 Like
5

Yes I get that but can you tell me the actual method on how to load an assets specifically? Like the “method”

6

Oh well you already know this with AsyncLoadClassAsset except instead of loading the whole data table you just load what you need.

So is your data table still full of hard references? if so you need change them to soft, then you can get the rows you need and load them individually

1 Like
7

Will try that. Thanks