Recursive calls to Nanite FStreamingManager::ApplyFixups and the intended use of bAllowReconsider

Development Rendering
, ,
1

(The following is a translation of a [Content removed] originally written in Japanese by Hirakawa Shoichi.)

Thank you for your continued support.,

I am trying to achieve specialized mesh rendering for one of our internal titles. For that purpose, I have forked the NaniteBuilder module and customized the build process so that it can generate Nanite meshes (Nanite::FResources) with Assembly Parts being instantiated across multiple Root Meshes.

I also have confirmed that the Nanite meshes can be loaded, streamed, and rendered successfully via our own custom assets and components, which are similar to UStaticMesh / UStaticMeshComponent.

Though I have customized the Nanite mesh build process, I have not modified Nanite streaming or the renderer. I am using the standard UE implementations for both.

The issue is, however, that when streaming some of the custom Nanite meshes with a large number of Mip Levels (and a high polygon count), the engine freezes.

Investigating the issue, I found that recursive calls to FStreamingManager::ApplyFixups continue for an extremely long time and appear to effectively never terminate.

Looking into the implementation of FStreamingManager::ApplyFixups, I found that the bAllowReconsider parameter does not appear to be referenced within the function as of UE 5.8.1. So, as a test, I made the following change so that the Reconsider other pages processing is performed only when bAllowReconsider is true. With this change, the freeze no longer occurs.

void FStreamingManager::ApplyFixups( const FFixupChunk& FixupChunk, const FResources& Resources, const TSet<uint32>* NoWriteGPUPages, uint32 NumStreamingPages, uint32 PageToExclude, uint32 VirtualPageRangeStart, bool bUninstall, bool bAllowReconsider, bool bAllowReinstall )
{
	// ...
	
	// Reconsider other pages
	#if 1 // EDIT_BEGIN
	if (bAllowReconsider)
	#endif // EDIT_END
	for (uint32 i = 0; i < FixupChunk.Header.NumReconsiderPages; i++)
	{
		const uint32 ReconsiderPageIndex = ResidentVirtualPages[VirtualPageRangeStart + FixupChunk.GetReconsiderPageIndex(i)].ResidentPageIndex;
		if (ReconsiderPageIndex != INVALID_RESIDENT_PAGE_INDEX)
		{
			ApplyFixups(*ResidentPageFixupChunks[ReconsiderPageIndex], Resources, NoWriteGPUPages, NumStreamingPages, PageToExclude, VirtualPageRangeStart, bUninstall, false, false);
		}
	}
}

So, could you clarify the following points, please?

(1) Was bAllowReconsider originally intended to be used to control whether ApplyFixups is recursively called for Reconsider Pages, as in the change above?

(2) If the above usage is correct, we are considering applying the same change as an engine patch on the project side. Are there any cases where this could cause issues with the consistency or correctness of Nanite Streaming?

(3) If checking bAllowReconsider alone is not sufficient as a condition for preventing recursive calls to ApplyFixups, is there a recommended way to keep ApplyFixups recursion to the minimum necessary while preserving the correctness of Nanite Streaming?

I would appreciate it if you could take a look at this issue.

[Attachment Removed]

2

Hi Masahiro,

1) Yes, that seems to be a simple mistake. However, as far as I tell, this should be benign and only affect performance, not correctness.

As far as I remember, the idea is that reconsider edges are always pointing to higher page indices, so the reconsider recursion should always terminate.

// Other pages in range reference the last page for reconsideration
if (PageIndex < LastPageIndex && !Resources.IsRootPage(PageIndex))
{
    PageFixups[PageIndex].ReconsiderPages.AddUnique(uint16(LastPageIndex));
}

2) No, adding the if(bAllowReconsider) should not affect correctness. I have tested it quickly locally and will try to get it into Unreal main unless I notice any obvious issues.

As far as I can tell whether the bAllowReconsider branch is there should not affect correctness. It should only be a (minor?) performance optimization. So the fact that you are seeing infinite recursion without it suggests that you end up in a situation where there is actually a cycle in the reconsider page reference graph. In that case the branch fix for infinite recursion is probably just treating a symptom instead of the underlying issue.

Do you have local changes that might introduce cycles in the reconsider page references? Also, if you are able to reproduce infinite recursion in mainline Unreal, we would be very interested in a repro.

I hope this is helpful.

[Attachment Removed]

3

[mention removed]​

(The following is a translation of a post originally written in Japanese by Hirakawa Shoichi.)

Thank you for letting me know that “if (bAllowReconsider)” does not affect correctness and only affects performance. This is also consistent with our test results.

I have not been able to reproduce the infinite recursion in ApplyFixups in Unreal mainline.

As for the behavior I observed with our customized Nanite resources, it appears to be infinite recursion at first glance. However, I do not believe that a cycle in the reconsider-page reference graph is the cause. We have not made any local changes that could introduce such a cycle.

Instead, I guess that the issue may be caused by a high-in-degree acyclic graph generated by the customized Builder. In this case, I am instantiating several thousand Assembly Parts with small to medium triangle counts across multiple Root Meshes. As a result, a single Page may be referenced for reconsideration by a large number of Pages with lower indices.

Even if the graph itself is acyclic, the current recursive traversal may end up following a huge number of different traversal paths, resulting in a combinatorial explosion in traversal time.

I will review the Nanite Builder customizations again, and also consider a local patch for the recursive reconsider processing.

Also, I’d appreciate it if you could let me know when the “if (bAllowReconsider)” change is integrated into the Unreal Main branch.

Thank you again for your support!

[Attachment Removed]