WorldGridMaterial load failure

Development Asset Creation
, ,
1

We are occasionally seeing the same error message as the thread [Packaged build is corrupted by unknown [Content removed] when trying to run cooked game builds from our build servers:

New pak file ../../../XXXXXXXXX/Content/Paks/XXXXXXXXX-WinGDK.pak added to pak precacher.

FlushAsyncLoading(1): 1 QueuedPackages, 0 AsyncPackages

Updated: NewPackages=0, OldPackages=0, TotalPackages=0

LoadPackage: SkipPackage: /Engine/EngineMaterials/WorldGridMaterial (0x953039A7B296B2EE) - The package to load does not exist on disk or in the loader

Failed to find object ‘Material /Engine/EngineMaterials/WorldGridMaterial.WorldGridMaterial’

appError called: Assertion failed: GDefaultMaterials[Domain] != nullptr [File:J:\w\XXXXXXXXXXX\Engine\Source\Runtime\Engine\Private\Materials\Material.cpp] [Line: 645]

Cannot load default material ‘engine-ini:/Script/Engine.Engine.DefaultMaterialName’ [Domain=MD_Surface] from path ‘/Engine/EngineMaterials/WorldGridMaterial.WorldGridMaterial’

While the symptom looks the same as that thread, the cause is likely different. In that thread’s replies, it mentions a corrupt Perforce workspace on the build machine. From our logs, it doesn’t look like there’s any consistency between what build machine(s) are involved, and other people downloading the same cooked build from the same source are able to load in just fine.

The following are the peculiar patterns I’m seeing in the crash database:

  1. It pops up every couple of weeks and then immediately disappears
  2. It only seems to be reported by external people (usually external QA, but at least once from a co-dev employee), never from anyone within the studio.
  3. It always seems to show up twice from the same person/machine/build about 2 minutes apart, then disappears until it comes back a week later for another matched pair from someone else.
  4. Most people downloading the same build from the same source are able to run the game without any crashes.
  5. Other than it always being someone external to the studio, there seems to be no consistency (like multiple QA people hitting it, but always on the same dev PC, or always coming from a specific build server).

Looking at other logs from the same builds which do not crash, at roughly the same place in the log, there’s this warning instead, which seems similar enough to guess that it might be related:

LogStreaming: Warning: Flushing package /Engine/EngineMaterials/DefaultMaterial while it’s being preloaded in the same callstack is not possible. Releasing request 6 to unblock.

I took a look through Perforce, and I found CL 43038808, which sounded promising as a potential fix for this, but I integrated those changes and I am still hitting the LogStreaming warning. The thread linked above is the only one I could find searching previous EPS/UDN discussions, but there may be something out there I missed.

[Attachment Removed]

2

Hello!

We addressed some problems in the way the default materials are loaded on startup in CL48392429. You might want to give that CL a try.

Regards,

Martin

[Attachment Removed]

3

You don’t need the other CL. I remembered that I provided the integration of that CL to a previous case. [Content removed]

You can retrieve the zip from my post in there. It should work directly as this was done on the 5.6.1 code.

Regards,

Martin

[Attachment Removed]

4

Are you still hitting the issue with the missing completion callback? If so, can you provide a log of the server loading hitting the issue when LogStreaming set to verbose? I’d be interested in confirming the loads that are occurring and the state of them.

If DefaultMaterials was implicitly loaded, not flushed due to a manual request then there won’t be a completion callback to fire until the manual request is made since that is when the callback can be fired. I’d be interested if you’re seeing an issue with that type of ordering or not (DefaultMaterial is already loaded but for whatever reason the callback is skipped when the request is made and completes immediately)

Also possible change to try out: 42844964 + 42893359 helped with initialization order for default materials and loading complications based on that varying order.

All the best,

Kev

[Attachment Removed]

5

Awesome, thank you.

[Attachment Removed]

6

While porting these changes over, I noticed this new code within UMaterialInterface::InitDefaultMaterials():

		// Explicitly initialize UMaterialInterface and UMaterial CDOs in this order to avoid issues with recursion and
		// async loading depending on initialization order of compiled-in UObjects.
		// i.e. when UE_WITH_CONSTINIT_UOBJECT is set.
		UMaterialInterface::StaticClass()->GetDefaultObject();
		UMaterial::StaticClass()->GetDefaultObject();

Looks like this comes from CL 45536001, which touches ~70 files. Do I need to pull in all of those changes as well for 48392429 to work? If not, should I leave these change lines out, or is there a subset of the larger CL that goes along with these particular lines that I will need to port

[Attachment Removed]

7

The common warning is gone with these changes…we’ll have to roll it out and wait to see if the other error is fixed as well! Thank you for pointing me to this CL/zip file.

[Attachment Removed]

8

So, this was working for Editor and Game builds, but we are having issues in dedicated Server build where one of the default material domains is trying to use “Engine/EngineMaterials/DefaultMaterial”, and the completion callback is never firing for that material package, but ONLY so far in Server build.

I’ve done some debugging and stepping through, and it looks like it is not being called because it is being loaded through a load flush for “Engine/EngineMeshes/Cylinder”, and is hitting a check where completion callbacks are only fired for the specific package being flushed in FAsyncLoadingThread2::ProcessLoadedPackagesFromGameThread()

It looks like this was fixed in CL 48035886:

- Changes the ordering for how package Completion callbacks are invoked. We now provide a guarantee that callbacks for imported packages will run before their importing package’s competion callback (this is also the case for circular dependencies). Previously, callbacks would only be invoked for the package being flushed exclusively.

So, I’m going to try to pull over that change and backport it into our engine version next.

[Attachment Removed]

9

Bringing over 48035886 did not help. The completion callback for /Engine/EngineMaterials/DefaultMaterial still was not executed, because /Engine/EngineMeshes/Cylinder is not in the CompletedPackageRequests array when GatherCompletedPackagesToFlush() is called. We’re loading and flushing RequestID 711 (Cylinder), but the only package in CompletedPackageRequests is DefaultMaterial (RequestID 9).

I’m guessing either I messed something up trying to bring the CL over that I can’t see, or else that CL depends on some other change in a previous CL that is not present in our build. I’m going to try attacking the problem from another angle.

[Attachment Removed]

10

I am on PTO this week, but I can get back to you on this when I’m back in the office.

We were only using DefaultMaterial in this case in the first place because the custom default material created for this purpose was not loading properly. I was able to figure out what was wrong in that case and fixed the asset. So, now we’re using a separate material that is not referenced by anything else but the default materials loading code, and everything is loading normally.

I can temporarily revert back to the previous version and gather some logs with and without those CLs for you when I am back next week.

[Attachment Removed]