Long cook times

Is there any work being done on multi-thread or multi-process cooking? That would be a huge win for us.

We currently have almost 28,000 assets that take close to an hour to cook. Is this within the ballpark of what you’d expect? We also noticed large increases in cook times from both 4.16 and 4.17 on all platforms (PC, PS4, Xbox).

We’ve added a lot of instrumentation to our cook to help track this down, but have not seen any smoking guns. Here’s an example of our cook. Do you have any recommendations on what we can look for?

Misc Cook Stats
===============
ShaderCompiler
    BlockingTimeSec=0.000000
    AsyncCompileTimeSec=0.000000
    GlobalBeginCompileShaderTimeSec=0.000000
    GlobalBeginCompileShaderCalls=0
    ProcessAsyncResultsTimeSec=0.004636
GlobalShader.Misc
    ShadersCompiled=0
MeshMaterial.Misc
    ShadersCompiled=0
MaterialShader.Misc
    ShadersCompiled=0
Package.Load
    NumPackagesLoaded=2212
    LoadPackageTimeSec=766.839767
Package.Save
    NumPackagesSaved=27784
    SavePackageTimeSec=1361.424483
    BeginCacheForCookedPlatformDataTimeSec=0.187994
    StripExportObjectsTimeSec=0.090898
    ImportObjectsAndNamesTimeSec=644.905822
    TagRelevantNamesTimeSec=0.549502
    BuildImportMapTimeSec=0.388614
    BuildExportMapTimeSec=0.510751
    SaveExportDepenciesTimeSec=61.549071
    TagPackageExportsPresaveTimeSec=262.638819
    TagPackageExportsTimeSec=247.364260
    ResetLoadersForSaveTimeSec=12.554244
    TagPackageExportsGetObjectsWithOuter=0.631885
    TagPackageExportsGetObjectsWithMarks=0.032586
    SerializeImportsTimeSec=0.000000
    SortExportsSeekfreeInnerTimeSec=39.703500
    SerializeExportsTimeSec=43.810249
    SerializeBulkDataTimeSec=3.597077
    AsyncWriteTimeSec=0.153850
    MBWritten=16628.566468

Cook Profile
============
0.CookWallTimeSec=3263.713983
0. 0.StartupWallTimeSec=344.205438
0. 1.CookByTheBookTimeSec=2919.506500
0. 1. 0.StartCookByTheBookTimeSec=20.913601
0. 1. 0. 0.GameCookModificationDelegateTimeSec=0.185849
0. 1. 1.TickCookOnTheSideTimeSec=2840.968260
0. 1. 1. 0.TickCookOnTheSideLoadPackagesTimeSec=564.124317
0. 1. 1. 1.TickCookOnTheSideSaveCookedPackageTimeSec=2070.635674
0. 1. 1. 1. 0.TickCookOnTheSideResolveRedirectorsTimeSec=0.000000
0. 1. 1. 1. 1.TickCookOnTheSideVerifyCanCookPackageSec=255.172285
0. 1. 1. 1. 2.TickCookOnTheSideConvertBlueprintsSec=0.752136
0. 1. 1. 2.TickCookOnTheSideBeginPackageCacheForCookedPlatformDataTimeSec=65.476262
0. 1. 1. 3.TickCookOnTheSideFinishPackageCacheForCookedPlatformDataTimeSec=0.811819
0. 1. 1. 4.TickCookOnTheSideWaitForAsyncWorkAndCleanUpSec=0.000044
0. 1. 1. 4. 0.TickCookOnTheSideAsyncWriteFilesSec=0.037170
0. 1. 1. 4. 1.TickCookOnTheSideSyncDDCFilesSec=0.009366
0. 1. 1. 4. 2.TickCookOnTheSideSaveShaderCacheSec=0.000000
0. 1. 1. 4. 3.TickCookOnTheSideGenerateManifestSec=0.000000
0. 1. 1. 4. 3. 0.TickCookOnTheSideSaveCookedAssetRegistrySec=0.000000
0. 1. 1. 4. 4.TickCookOnTheSideMapDependencySec=0.000000
0. 1. 2.TickLoopGCTimeSec=0.000000
0. 1. 3.TickLoopRecompileShaderRequestsTimeSec=0.000978
0. 1. 4.TickLoopShaderProcessAsyncResultsTimeSec=0.000800
0. 1. 5.TickLoopProcessDeferredCommandsTimeSec=0.002197
0. 1. 6.TickLoopTickCommandletStatsTimeSec=0.000184

We also have lots of ActiveClassRedirects and other redirector. Do these affect cook times?

Thanks, Joe. We regularly fixup content redirectors because we knew those would be a problem.

Try this out: https://udn.unrealengine.com/questions/403509/asset-loading-poor-performance-on-windows.html

There were some regressions in cook time performance over the past few versions, but we’ve recently been looking at them and have made a few fixes that have helped. These changes should all be in our main branch now and should be available in 4.19. If you need some more specifics about cherry picking things that might help, let me know.

Otherwise, I’d be double checking the usual suspects. Are you happy that the assets that are being cooked are all required for your runtime game? (i.e. are you cooking test data or editor only reference data that isn’t necessary for the game?) Do you have a shared DDC server set up to cache intermediates? Have you tried iterative cooking to speed up incremental builds?

If you want to attach a full cook log, there might be a bit more information in there.

I had a quick look over those logs and compared it with some equivalent Fortnite logs. Everything kind of looked similar, so I don’t think you’re a million miles away from what we would expect. I noticed that there are a few minutes taken up by that verifycancookpackage stuff that I mentioned above, so you could save a few minutes by removing that?

Someone else will have to weigh in for the class redirects. For content redirects, they will increase cook time for each content, slightly, since it requires a load, get a new path to the content, and a final load

Thanks, Riley. That’s a useful change, but it doesn’t look like that code path is taken during cooking.

Here the timings we have:
4.12.5:

UE4Editor-Cmd: Execution of commandlet took:  339.50 seconds

4.17.2:

Cook: Execution of commandlet took:  3846.86 seconds

Around 80% of time goes to GEditorSavePackage. Is that something we can expect to be addressed in 4.19?
link text

Looking at your log, that particular run of the cooker did a lot of DDC generation. What kind of timings do you get when you re-run that cook a second time using the primed DDC?

The changes we have made so far that will appear in 4.19 have helped a lot with our own internal project cook times, although they are still slower than several versions ago. It’s difficult to say whether they will “fix” your problems, but they should hopefully improve things.

Cheers,

Graeme

Archiving just creates a complete copy of the build output in a specified directory, and shouldn’t affect cook times noticeably.

It might be worth having a look at this post, as they were reporting similar issues to you and managed to get some good improvements!

https://udn.unrealengine.com/questions/397467/cooking-started-to-take-much-longer-time-after-upd.html

The basic gist was that the VerifyCanCookPackage function on the asset manager class can be slow, and if you’re not using the “never cook” or “development only cook” functionality then there could be a win by deriving your own asset manager class and overriding that function to disable the checks within. The licensee in that linked thread reported a big win from doing that!

link text

This is a fairly fast iteration of our cooker, cooking only maps that we are actively testing. We get 97.5% hit on the local DDC, but still spend a lot of time saving the 13GB of packages.

I had a quick look over those logs and compared it with some equivalent Fortnite logs. Everything kind of looked similar, so I don’t think you’re a million miles away from what we would expect. I noticed that there are a few minutes taken up by that verifycancookpackage stuff that I mentioned above, so you could save a few minutes by removing that?

Thanke, Graeme. We have disabled VerifyCanCookPackage, and it did help. We’re also doing our best to limit the number of maps we’re cooking. One thing we found that made a huge difference was adding memory to our build machines. 64GB just wasn’t enough, and 128GB works much better!

Awesome. Yeah, more memory is a great idea as it will reduce the need to collect garbage, which can flush out loaded assets that will need to be loaded again later on!

And faster NVMe drives also help quite a bit.

Yep we have that change but our cook times are still high.

The timings when the cook ran the second time is pretty much the same. I would guess the cooker is using the cache. Though I assume it does it by default, we don’t do anything specific for that rather than using the same machine for the same configuration.
That’s the arguments we use to cook development build:

-noP4 -platform=Win64 -config=Development -game -build -cook -allmaps -stage

The arguments are copied from :

A new, community-hosted Unreal Engine Wiki - Announcements - Unreal Engine Forums

With the exception to archive part. The argument didn’t have any effect on time in 4.12.5. Let me know if it’s recommended to use the archive for cooking.

I seem to have no access to the post. Could you please share the details that might be useful?