After world partition has some cells from our heavy area added to world, we see some huge bumps from the FPBDRigidsEvolutionBase::FlushAsyncAccelerationQueue which called FPBDRigidsEvolutionBase::ApplyParticlePendingData for 10k+ times and caused the gamethread to wait for it on TG_EndPhysics.
[Image Removed]
[Image Removed]
[Image Removed]
We see there are some other EPS members have faced similar issues, and tried to
- Change PendingData.RemoveAtSwap(SlotIdx, EAllowShrinking::No);
- Tweak p.Chaos.AccelerationStructureCacheOverlappingLeaves and p.Chaos.AccelerationStructureTimeSlicingMaxQueueSizeBeforeForce
But obviously they are not the main catch of the too many calls to FPBDRigidsEvolutionBase::ApplyParticlePendingData and the hitch still persists.
The questions:
- What’s the difference between the work done in FlushAsyncAccelerationQueue and FChaosAccelerationStructureTask? Why can’t the FlushAsyncAccelerationQueue be async or time sliced?
- How to eliminate such hitches? We have made some tools to merge collisions into trimesh to reduce the total number of body instances to get streaming in. But we only expect the artist could cut half of the number. So we’re wondering is there any other solution?
- Can Chaos and World Partition built AccelerationStructure for static objects of each cell offline?
[Attachment Removed]
Note: All the traces of the above screenshot came from editor, but we also noticed similar hitches in the packaged game.
[Attachment Removed]
Hi, and apologies for the delay.
For 1. The 2 jobs do different things, and the context will help here.
The FChaosAccelerationStructureTask iterates through the current snapshot of the particles (at the moment in time that it started) - it may take multiple frames to do that, and can be time sliced since nothing is actively reading it. This produces a more balanced tree than if we just iteratively added the changes to the broadphase over time (add/remove/move). During that time there will be alterations happening which it won’t have existed at the moment this build started.
The FlushAsyncAccelerationQueue is the answer to that and is triggered when this rebuilt tree becomes the current broadphase- ie "The tree has been rebuilt for frame N, but we are on frame N+5 now - so flush all the operations from N to N+5 to the tree. If we don’t do that all at once, the collisions won’t work since the particles will be in the wrong place. This is why this part can’t be timesliced.
There are 2 reasons then why the ‘flush’ can be large and blocking - either there are too many particles being altered within frame, or the rebuild took too long.
Looking at the data you’ve provided - it could even be both in your case. Do you have an idea of how many particles you have? I can see that the flush has over 10K operations which is HUGE for what we’d expect
[Attachment Removed]
For 2 - Reducing the number of bodies is very much the main way to go here. What is the limiting factor for this - and what is the breakdown of static and dynamic bodies in this case? - ie all static bodies should have the same position relative to each other right?
For 3 We do not - but it is something which we have considered as well. Generally the main path is to reduce the static bodies to a level where this isn’t needed.
Hope this all helps!
Geoff
[Attachment Removed]
Hi there,
“If I understand correctly, the FChaosAccelerationStructureTask just optimizes the unbalanced tree which was already updated on game thread?” - This could be read in a few different ways, so lets clarify so we are aligned! Yes, the FChaosAccelerationsStructureTask balances out a potentially unbalanced tree. Updating this tree on the game thread can happen - but it is overloaded so it depends on the implementation. The gamethread tree would be unbalanced if all of the bodies were pushed to it, and would essentially mimic the latest version of the current physics tree. (These trees are buffered).
The questions you ask afterwards lean towards avoiding the rebuild work until the level is fully loaded. We don’t have an obvious mechanism for that, and honestly - I’d still stick with the same overall point that you’d be addressing the wrong issue here.
The best way to solve this issue the usual way is to both reduce bodies, and the geometry. For the modular castles etc - are they runtime generated or are they essentially prefabricated? Does each body only have one shape? Is the landscape made of one large mesh, or discrete components?
Best
Geoff
[Attachment Removed]
Thanks for the answers 
This is something we have fairly frequent conversations about with clients. Once we have a prefab sorted, you can merge the shapes onto one body (which alone will reduce the cost) - so 1 castle = 1 body - and from there we have the modelling tools in the editor which can merge the meshes themselves, and reduce the collision complexity. The collision meshes will nearly always be better quality when they are not hugely complex. If you do both of these you should find a huge reduction in that volume. For the landscape, all static objects can be baked down into a series of trimeshes to reduce complexity here as well.
From your description, it is also worth really hilighting that anything which the player ‘interacts’ with can just be a trigger box, and anything that a player can’t collide with, or interact with doesn’t need a collision geometry at all.
If this is a fairly large world - it could also be a decent shout to use world partition, and lightweight actors if you aren’t already - again this will reduce the load down substantially!
Best
Geoff
[Attachment Removed]
Hi Jenkins,
No worries. I’m not following why this would lose the flexibility of modularity though - there are prefabricated at compile time right? Generally a lot of these processes can be automated in some way - this kind of thing is what Tech Art tends to excel at. We are getting more support for what are ‘non destructive workflows’ built out as well.
Can I check what the situation is with the situation we are talking about - I’m assuming you are loading a level? - is this at the start of gameplay or during it? Could pausing the physics be a potential solution here?
Best
Geoff
[Attachment Removed]
Hi Jenkins,
A lot of what you are looking at we had to also solve when we did our ‘city sample demo’. https://dev.epicgames.com/documentation/unreal\-engine/city\-sample\-project\-unreal\-engine\-demonstration
For this project, the modular meshes were kept, but the collisions were turned off. Then we generated a specific collision mesh as a separate component which had no visual. This gives the flexibility of reusing instancing of the visuals, but without a huge cost for bodies. This also used world partition, and from memory - we could navigate at pretty high speeds without hitching (and we’ve since done more optimization since this project).
It could be worth taking a dive into that project for inspiration and to see how that setup looks.
The other potential route is to use dataflow or geometry scripting. The dataflow one has an example in the ‘ContentExamples’ project in the Physics_Destruction map on the Convex Generation section which should help.
Which ever route you take, this along with world partition should resolve the issues you are seeing.
Geoff
[Attachment Removed]
Hi Geoff, thanks for the very detailed explanation!
For the particle count, we have about 45k+ bodies, a huge number I know, however we have really complicated scene (castles, towers, rocks) composed by modular assets where player can get enter into them somehow, so it’s hard to reduce the unnecessary static bodies to the level.
[Image Removed]
If I understand correctly, the FChaosAccelerationStructureTask just optimizes the unbalanced tree which was already updated on game thread?
If so, can we abort the async rebuilt task if the num of changes in FlushAsyncAccelerationQueue exceeded a certain number?
Or can we disable the async rebuilt task, and manually trigger it when the level streaming state is stable?
[Attachment Removed]
Hi,
“For the modular castles etc - are they runtime generated or are they essentially prefabricated? Does each body only have one shape?”
They are prefabricated and each body only have one shape.
And those buildings have very dense inner decorations which have collision even though the player is currently far away from them, the player can see them but temporarily can’t interact with them.
The landscape is made of the landscape system but decorated with numerous numbers of discrete components (small rocks, debris, etc)
We tried to push the artists to minimize the amount of necessary collision bodies, but without much luck, at least for some areas of our level, there are too many densed and necessary meshes with collision.
“The questions you ask afterwards lean towards avoiding the rebuild work until the level is fully loaded”
We might still get interested in how to avoide the rebuild work until the level is fully loaded, if no effective way to reduce number of bodies.
[Attachment Removed]
Thanks for the suggestions
We will continue to reduce the bodies count via merge and disable unnecessary collisions. But this extremely slows down the development iteration speed, and lose the flexibility of modularity.
However from the engine side, I still argue with the current implementation.
If the AsyncAccelerationQueue grows very large after the async structures get optimized/rebuilt, then applying all those updates incrementally to the newly built static structures can become a bottleneck and make the tree unbalanced again.
So I thought there could be an option to abort and relaunch the FChaosAccelerationStructureTask if the AsyncAccelerationQueue is larger than a threshold.
And I want to figure out how to achieve this 
Best
Jenkins
[Attachment Removed]
Hi Geoff,
“Generally a lot of these processes can be automated in some way”
Currently we only have manual tool like merging bodies into trimesh (then remesh and simplify) per PackedLevelActor. And tool utilize the modeling tool which allows to generate and edit single trimesh for large ground collisions.
We have also considered about using RuntimeCellTransformer to merge collisions per cell, but it exceeds our memory budget, so we decided to let the artist choose which to merge instead of automating all of them.
May I ask what non destructive workflows are suggested here?
" I’m assuming you are loading a level? - is this at the start of gameplay or during it? Could pausing the physics be a potential solution here?"
We have a large open world with very densed meshes (200k instances over 4km*4km area). And these hitches happened during gameplay while the player runs from one place to another.
We also use world partition, but again back to the issue, there are some cells contain too many bodies and the player is traveling around quickly, which causes the AsyncAccelerationQueue becomes extremely large.
Best
Jenkins
[Attachment Removed]