I’ve been getting back into UE5 BPs recently; I’ve been reading a lot about Data Assets, and I’m worried about my current Inventory System.
Here is how it works for context:
The Inventory is a component, attached to the PlayerCharacter. Inside the component is the logic and data for the inventory.
The various data is stored in the Inventory as an arrays of structs, one for Items (ItemStruct) and one for Weapons (WeaponStruct), as weapons have more information needed (like firemode).
PickUps (for both Weapons and Items) are Objects containing a Struct, said Struct is composed of the RowName ID and the non-static Data (content, durability, or firemode, ammo in the case of weapons)
Upon picking up an item, the InventoryComponent gets the RowName ID and gets the Row from a DataTable (DT_CommonData) containing all Static Data shared by everything the inventory can fit (information like the name, the weight value, the item size and volume it takes in the inventory). Using that static data from the DataTable, the InventoryComponent checks where and if the item can fit, and if the function returns success, the Data from the Struct in the PickUp is added in the corresponding Array, and if it’s a success, the PickUp is destroyed.
Dropping an item is the reverse operation; spawning the right PickUp, specified in the DataTable, setting the Data, and removing the index from the Array in the InventoryComponent.
When an Item or Weapon is ‘equipped’, the InventoryComponent gets the DataTableRow from the ID in the appropriate DT (DT_ItemData for Items, DT_WeaponData for Weapons) which contains the information to initialize the Item through an Enum; for instance “is the Item a container/Food/Tool?”. Everytime the player equips an Item or Weapon this process happens, fetching the static data from the matching DT.
After reading a lot about how Data Tables and Data Assets work, I’m under the impression that my Inventory System isn’t all that great performance wise, and I’ve failed to find a proper answer, let alone a solution.
I’d be grateful for any clues, advice or information regarding these questions I have:
I have read that everytime a DataTable is called, the whole is loaded; knowing that my DataTables will contain hundreds of Rows at the end of the project, it sounds harmful to performances; should I consider another solution for getting and storing my static data? Are Soft References a serious way to address this concern?
I’ve read that Data Assets are great for storing non-static data, should I abandon Structs as a means to store my non-static data for items and weapons?
I want to make my project scalable and unburden my future self with complete reworks of systems, but I feel I don’t understand how Structs, Data Tables and Data Assets work quite yet!
To be honest, I think your is the best approach there is. Yes, the whole data table is loaded first time it is accessed but I see this as a perk, not a problem. You can have the names and icons of all items readily available throughout your game. The table itself should be manageable even if you have thousands of entries. (unless the structs are truly gigantic)
That being said there are several things you can do to optimize that.
Make sure your structs are light and with fixed size.
Store copies of those lightweight structs both in the inventory and in the pickup (instead of just the id) This will minimize DataTable look-ups and traversal. (although finding an item by id in a sorted array is lightning fast)
Soft references are useful to hold data that does not conform to the “light-fixed size” struct. You should keep the easy and useful parts of your items in the DataTable and large and unique parts (meshes, animations, 4k textures) as soft reference to load only when needed (lets say when equipped).
Data Assets are useful in the following scenarios:
You work with a large team and everyone working in a single DataTable is just not manageable.
You have smaller number of large objects.
You want to control how the assets are packaged.
You want to stream your data because it is too large.
P.S.
I forgot to mention that this is not a clear cut case and there is a lot of opinion involved. You can read all day about DataTable vs DataAssets but, in the end of the day, you can use both of them with very similar effect.
this is kind of false, yes you can modify the data but you’re doing it to the baseclass which effects all instances. ie if you changed durability all items would change.
i personally prefer DataAssets, the big advantage is they support inheritance so you can have the one item class with subclasses for weapons, armor etc.
that said dont change your whole project if its already working
Yes, if you use any one row in a data table, then the entire table is loaded.
Assuming that you use soft asset/class references for the “what is this thing?” fields, then that is unlikely to even show up in a profiler.
A single mesh for a single flower will probably use a lot more memory than this table.
I would recommend against mutating Data Assets. Store mutable data on an actor or component of some sort, or as a struct in some container that in turn is stored on a component or actor.
And, yes, soft references is the way to improve memory behavior of large systems. They don’t load assets into memory until you resolve them. Of course, this means that you may get a small hitch/glitch when you first use some particular item during the game session. If your game is a single-person development, you will probably not be able to develop enough content that a gaming computer couldn’t just keep it all in RAM, so you might be better off just loading everything and calling it good.
The only answer to “is bad for performance?” comes from a profiler. Run the game, on the target hardware. Measure its performance. If performance is good, the thing you’re doing isn’t bad for performance.
So far the struct for the DTCommonData contains roughly 20 variables, half being Integers and Booleans, but some being 2D Textures (icons), Text and Enums, is that veering towards a non-light struct? I’m unsure as to what constitute lightness in a Struct, but I’d assume it’s “few and simple variables”?
Regarding Struct Copies; the way my WeaponComponent works is on Equipping/initialization of a Weapon the data is fetched from the appropriate DT and then stored in the Component, because I had a bad feeling about getting the DT row everytime the Weapon was used to get its Base Damage, there will be fast firing weapons, so I thought it’d be better to save these values directly.
I could get that data from the PickUp and store it in the Inventory upon succesful Pickup if it’s better performance wise?
I think I misunderstood what you meant when you mentioned “keeping the easy and useful parts of items…”; should I remove the meshes and animations from the DT? Would I move them to another DT or is there another way to know exactly which mesh/animation an item would need without a DT?
Thank you a lot for your insight, you have already lifted a huge weight off of my shoulders, I am grateful!
Would you use Data Assets for, let’s say, Sub-Types of Weapons Logic? For instance a Baseclass “Gun” with a few subclasses like “pistol”, “automatic” and “shotgun”?
I am interested in learning new tools, so even if it’s not the answer for the InventorySystem, it could still make life easier for another part of the project!
Thank you a lot for your input, I really appreciate it!
its possible, the answer really lies in what your game needs.
too much inheritance can get problematic too, i’d probably just have a baseclass ‘item’ with subclass ‘rangedweapon’
but really for indie games its not too important as long as you know what you’re doing.
I’m unsure as to what constitute lightness in a Struct, but I’d assume it’s “few and simple variables”?
The “best” structs contain plain data only with fixed size. The main goal is to align the data in memory and prevent cache misses. I’m don’t feel competent enough so I won’t even try to explain these topics. Here is some info if you want to learn about that:
I could get that data from the PickUp and store it in the Inventory upon succesful Pickup if it’s better performance wise?
I doubt you will see any difference if you don’t make it constantly (on tick) but having fewer redirections is always faster. The downside is you fill your memory with redundant data but from the topics above you can see that data locality is actually more important.
should I remove the meshes and animations from the DT?
They are probably stored as references there anyway but yes. I would make them into an object/actor and leave a soft reference in the DataTable to it. Next I’d load the soft reference only when the object is taken by the player. These could be soft references to Data Asstes even
I’m sure there’s some kind of micro-benchmark where this matters.
Also, if you’re building a massive multiplayer game with a thousand people all firing automatic weapons at the same time, this may matter.
However, it feels to me as if you’re way over-optimizing something that probably doesn’t matter, and doing it way before it’s time to worry about this level of optimization.
This is the kind of optimization that was meant when Knuth said “premature optimization is the root of all evil.”
It’s true that choosing the correct algorithm is important! Scanning through every actor every frame to find something, when you could just have cached a reference to that something, is a bad algorithm, and should probably never be used in the first place. Selecting the right algorithm, is fine design work, that you can and should do early. But, optimizing struct sizes for cache behavior, only matters if you have literally thousands of them, and scan them all every frame.
Optimizing the struct layout for each particle in a particle system? Abso-freaking-lutely!
Optimizing the layout of the inner loop of a DSP function for a procedurally generated sound? Probably a good idea!
Shaving one cache miss off of something that maybe happens five times a second? Nobody will notice the difference, on any machine that is at all capable of running Unreal Engine. Even if you fail to do this optimization ONE HUNDRED DIFFERENT TIMES and it happens EVERY FRAME, you won’t notice one hundred cache misses per frame. The cost of one cache miss is 0.08 microseconds. The cost of one hundred cache misses is 8 microseconds. 8 microseconds at 60 fps is 0.03 fps difference.
As long as you choose a path and stick to it, you’ll be fine. The two paths are:
You’re a one man band. You will never be able to develop enough content that a modern computer will worry about it. Load it all in RAM and call it good! Ship a game, be happy!
You’re extremely worried about initial load time, so much so that you change all your asset references (icons, animations, sounds, …) to be soft references, and load them on demand. Your game loads slightly faster! Ship a game, be happy!
Also, the way that sub-classes are done in Unreal is generally by creating blueprints of the subclass. Each blueprint has the customized ammo/fire-rate/sounds/meshes/icons. An actor blueprint is a class, quite physically. Thus, you develop all your guns as blueprints that derive from your “Gun” base class. (Which may also be a blueprint, or a C++ class.)
Then, in inventory, you have a struct that has a soft class reference to the blueprint of the item. The inventory struct should also contain a soft asset reference to the icon to show in inventory, and the user visible name as string. You don’t need to add parameters related to the gun behavior, unless you want to show all of that in the inventory, too.
It’s true that I’ve been stuck on this for a while now, because I’m scared of shooting myself in the foot in the future, while oblivious that I may be doing it to my present self!
Do you feel I should still go ahead and have the values saved in the PickUp → Inventory Component upon picking up? Or get said data during item initailization from the DT?
I don’t plan for thousands of players, it’s a solo game with, hopefully, possible co-op, but I do want to set the ground up properly for a standard PvP / be able to reuse my code for later projects.
Thank you for putting this issue into perspective with the microseconds/frames comparison, it helps a lot! I guess the answer is optimize when needed afterall, as much as I dislike it.
Soft class reference to the blueprint of the customized/specific weapon? That sounds like a good idea, I’ll try that!
Thank you very much for your answers and insights!
your system is fine but honestly, the absolute simplest way to do it is just have 1 actor per item.
you can
set/get the default values in the class defaults.
Instantiate the dynamic values in your spawned actor.
reference them all in a DataTable with softclass references, or just use the asset manager.
actors can replicate if you go down that path
you can self reference in inventories (ie items within items, you cant do this with structs)
there are more advantages too, yes an actor is technically the heaviest class but for simple indie games i cant see it being a problem and the speed/simplicity saves development time
I’ll keep experimenting with the data assets / actors for the static data
One issue/worry of my current set-up is unused variables for weapons; some weapons have logic that isn’t shared with most:
Removable/switcheable barrels
Removable Stock changing fire selector options
Auxiliary weapons
Etc.
So making individual assets/Actors bundled with the mesh, animations, etc. seems like a good way to implement what is suggested here and should also remove me the headache of the special cases dictating the content of Structs for the rest.
On creation of the PickUp Object, the DT(s) is called to set both the Mutable Data (mutable is non-static if I understood right) and a Soft Reference to the specific Blueprint, which has the Static Data
On successful Storing of the Item, the two are added, with the Mutable Data being ordered in the appropriate Array (only need 1 set of static data for X instances of identical items (mutable data array) in the inventory)
This should solve:
Excessive DT look-up during gameplay
PickUps holding data and logic that is only used when in a player’s possession
Unused variables only required by specific items/weapons
I’m not experimented but trying to figure things out and experimenting is a really fun activity for me!
PickUp had struct with mutable data and RowName ID
Items/Weapons were stored as Arrays of said Structs
Upon equipping Item/Weapon (putting in one of two hands) the DT was called (Get DT Row) and the data was initialized, which meant setting the Data from the appropriate DT
When opening the inventory, the Icon, Description, etc. Was fetched from the DT, which meant getting the same icon 10+ times if the players had 10+ times the item
I didn’t feel it would hold up to being scalable, especially with hundreds of Rows and inventories capable of holding hundred of items
I know this has been solved, but I’d still like to add a few things.
If this is for single player then having Inventory in the Pawn or Controller is fine. If multiplayer it is not. Consider network issues and the player gets dropped. The Pawn and Controller on the Server will be destroyed along with any data those actors and components are storing.
Player State on the other hand will persist on the server a bit longer… long enough to copy & store needed data. Thus anything that needs to persist in the event of a player disconnect (client crash, internet issue etc) should be stored in the player state.
I use Game Play Tags to identify interactive actors. For pick up items I use a 3 tier system. World Item . Item Type . Specific Item … [WorldItem.Weapon.M416]
I also use Data Tables. For each Item Type I have a separate table for faster reading and to reduce load. [Weapons, grenades, ammo, healing, attachments, gear etc etc]
For any item that’s persistent in the game world such as gear and weapons (spawned and attached) I store DT data in the actual actor/component.
For example when I pick up a weapon the world actor (simple variant) is destroyed and the server spawns and attaches a high level variant to the pawn. On spawn the weapon configures itself based on the data table information.
I Parse the GamePlayTag to return a Weapon ID (name) for DT lookups. For weapons I have multiple data tables. Base Config, UI Data, FX (Visual/Audio), Ballistics (muzzle velocity, spread, recoil etc) and so forth.
Here are a few of the look ups. Note I’m soring the row in a struct in the actual weapon class.
Solved or not any extra information/insight/advice is always welcome!
I’ll look into Player State and GamePlayTags!
I have a few questions:
Is the BuildConfig Function called during gameplay? “Spawn” has me confused
Any reason why you prefer item Static Data to be in DTs compared to Weapons?
Let’s say your player uses an item, does the function that uses the item already have the Static Data, or does it gets it from the appropriate DT or DTs?
In my current system two DTs are accessed when using an item; the Item Data DT and the specific Item Data DT: the first says if the item is a medical item for instance, while the Medical Item DT (Specific Item) has all the relevant Data, is that similar to how yours work?
Regardless, thank you a lot for your time and insights!!
BuildConfig runs on Begin Play in the weapon class when it is spawned for both the Server and Client. Simulated Proxies do not run the full build. A separate one just for FX data is used.
The weapon parent class is essentially a husk. The core logic is stored in actor components and the DT stipulates which shooting abilities the weapon can have.
The end road for my weapon class is to have a single weapon actor class that can be configured to be any weapon simply by assigning it a gameplay tag. The config will set the mesh, cosmetics etc. Essentially a dynamic data driven weapons class.
I prefer data tables because it makes tweaking specs a lot faster vs opening classes. You can even do run-time data table modifications. I can store all specs on a backend server and simply pass them off to the server and clients as they join the server. No need for patches.
When the item is replicated to the client and spawned locally the begin play buildconfig will execute. So it gets and sets class variables (structs) at run time.
When the client uses the item there are no further DT calls.
Imagine firing an automatic weapon and having to do look ups for each shot. That’s an FPS killer in itself.
