Cooking fails with multiple CookWorkers because max packet size is exceeded for a single message

Development Programming & Scripting
,
1

We are attempting to enable multi process cooking by increasing the CookProcessCount setting, but unfortunately we get crashes like this one when cooking:

LogCook: Display: LostConnection to CookWorker 3. Log messages written after communication loss:
LogSockets: Error: [CookWorker 3]: Could not WritePacket to Socket. A single message with Guid 4DF3B36BBA2F4E04A846E894E24EB2C4 is larger than MaxPacketSize 1073741823.
LogCook: Display: [CookWorker 3]: Cooked packages 9809 Packages Remain 141 Total 9950
LogCook: Display: [CookWorker 3]: Cooked packages 9827 Packages Remain 123 Total 9950
LogSockets: Error: [CookWorker 3]: Could not WritePacket to Socket. A single message with Guid 4DF3B36BBA2F4E04A846E894E24EB2C4 is larger than MaxPacketSize 1073741823.
LogSockets: Error: [CookWorker 3]: Could not WritePacket to Socket. A single message with Guid 4DF3B36BBA2F4E04A846E894E24EB2C4 is larger than MaxPacketSize 1073741823.
LogSockets: Error: [CookWorker 3]: Could not WritePacket to Socket. A single message with Guid 4DF3B36BBA2F4E04A846E894E24EB2C4 is larger than MaxPacketSize 1073741823.
LogSockets: Error: [CookWorker 3]: Could not WritePacket to Socket. A single message with Guid 4DF3B36BBA2F4E04A846E894E24EB2C4 is larger than MaxPacketSize 1073741823.
LogCook: Error: [CookWorker 3]: CookWorkerClient failed to write message to Director. We will abort the CookAsCookWorker commandlet.
LogCook: Warning: CookWorkerCrash: CookWorker 3 failed to read from socket, we will shutdown the remote process. Assigned packages will be returned to the director.
(This is followed by LogCook: Error: Package [redacted] can only be cooked by a now-disconnected CookWorker. The package can not be cooked. for each of the remaining packages assigned to the CookWorker.)

We are on UE 5.4.3. On every occurrence it fails on the same GUID, which leads me to believe there is possibly some asset which is causing this problem. For now, we have managed to work around this particular issue by increasing MaxOSPacketSize in CompactBinaryTCP.h (removing the “further restriction” which is explained in the adjacent comment). We do not consider this a permanent solution though, and would much rather figure out if there is something we can do to our game data to properly fix this problem (and prevent it from happening again).

2

Hi @snarkussharkmob
Let’s see…

Identify the Faulty Asset: Locate the asset that belongs to the GUID causing the game to crash and confirm its size, complexity, and dependencies.

Optimize the Asset: If the asset is complex or huge (e.g., large textures, high-poly models), optimize the asset by compressing textures, reducing meshes, or splitting large assets into smaller parts

Investigate Packet Size: Even though you’ve increased MaxOSPacketSize, don’t use this as a temporary solution. Instead, try to reduce the asset size or serialize data more efficiently when cooking.

Check Cook Settings: Ensure cook optimization settings such as chunking are enabled to handle large assets better

Update Engine and Plugins: Verify if newer versions of Unreal Engine or any of the plugins you’re employing solve the issue

Test Remote Worker Configurations: Verify remote workers are well-provisioned and network stability is not causing communication breakdown.

Optimizing assets and inspecting cooking and network configurations should fix the problem for you without your having to go back to raising packet size limits

3

We would prefer to fix the asset - however, the GUID does not correspond to an asset but to a message type, in this case FShaderLibraryCollector (the way I understand it). Hence, we cannot identify the asset by the error message.

What you’re saying about cook settings is interesting though. Could you provide some specifics on the chunking setting? I couldn’t find it in the ini files.