With a large open world map, I am hoping to use partitioning so that if an artist makes an update, such as a landscape sculpt, in a small area, the entire PCG Biome system doesnt need to cook.
Just doing so on the Biome Core doesnt serve this function, since that is just gathering the data from the other actors, so we must set up partitioning on every Local Biome Core
The time to generate goes up exponentially. It goes from almost instant to taking close to a minute, with a progress bar that claims over 200k PCG tasks.
It looks like multiple operations might be getting called redundantly(more so that just 1 for each partition) but its difficult to troubleshoot.
Or is this just the extra cost of partitioning, since the data needs to be initialized separately for each individual partition?
It does appear to work as desired after that long initial cook, but even then it takes as long or longer to cook just those updated partitions.
[Attachment Removed]
Steps to Reproduce
Open biome sample level (5.7.2)
-On biome core, set the PCG component to partitioned
-On each individual PCG biome actor (the volumes, textures and splines), do the same
-On the PCG World Actor, set partition size to 12800
-After the EXTENSIVE cook time to generate the foliage, sculpt the landscape in a small area
[Attachment Removed]
Hi Karl,
So this is partially because when any of your partitions of the local biome cores change, it will trigger a “full” change up to the global biome core(s), so that might cause more impact.
We’re trying something to alleviate a part of that in 5.8 - I’ll let my colleague give you a link to it when it’s in.
Cheers,
Julien
[Attachment Removed]
Hey Julien
It makes sense that it would function that way when you have just Biome Core partitioned.
My question is why enabling partitioning on the local biome cores causes such a massive change in the cook time and PCG tasks.
Is it just the inefficiency of doing those initial operations per partition as opposed to doing them once across the entire map?
Thanks!
K
[Attachment Removed]
Hi Karl! Sorry for the late answer.
There are multiple things here:
- When partitioned, the same graph is executed and dispatched to all the local components. Local Biome Core is not optimized for partition, meaning that there are probably a lot of things that are done for each local component that are redundant (pulling data from other actors, reading the assets etc…), this by itself can make the number of tasks skyrocket (which is not necessarily a bad thing, a lot of tasks is fine as a lot of them can be multithreaded and evaluated in parallel).
- What you are doing in the custom graph you can provide to the system (like the Local Biome Cache graph, or generators) need to be aware of partitioning. For example if you do a “Create Points Grid” somewhere, this will be duplicated for all the local components too, so you need to make sure that the points you generate are culled to within the local component bounds (with a Cull Points outside Actor Bounds).
- Until very recently, we didn’t have a smart refresh system with PCG Component dependencies when both are partitioned. When a small change (like landscape painting) would refresh a local component of PCG Component A, we were notifying PCG Component B that something has changed using the original bounds of component A (instead of the bounds of the local component that has changed). If the bounds of Component A and B were covering the full map, a small change in A would make B fully re-generate. This is improved in 5.8 where we are adding the bounds of all the local components that have changed and notify B with those bounds instead.
- Finally, I encourage you to watch our GDC talk where we try to explain how to do things at scale, what are the drawbacks and what you can to to mitigate them: Developing Large Procedural Systems with Low Friction and Fast Generation (Presented by Epic Games)
If you have other questions, don’t hesitate
Adrien
[Attachment Removed]
Thank you. That video had a few gems in it that I had not come across yet, mostly the Preview mode and the PCG build asset. We had recently moved to a dedicated build server and were suffering issues with artist checking out unneeded PCG partitions. This is a godsend for that.
At the end of last week I had come across how the local biome core was handling partitioning, in that generators were being processed not by partition, but for the entire area of the pcg component every time, and then being masked by the partition, which is a huge redundancy.
Either way it sounds like 5.8 is what I need to wait for in order to get that smart update within PCG partitions
[Attachment Removed]