We have at least two levels (both of which happen to have World Partition & r.AllowStaticLightingInWorldPartitionMaps enabled -- not sure if that’s relevant or not), which are soft-locking while waiting for …
bool bAllTaskAreComplete = (FPlatformAtomics::AtomicRead(&NumCompletedTasks) == NumTotalSwarmTasks ? true : false);
… to become true in FLightmassProcessor::Update().
However it never becomes true because the number of reported-complete tasks (NumCompletedTasks) is actually greater than the number of tasks issued (NumTotalSwarmTasks) by the end. (Presumably, the Update poll doesn’t happen to catch the instant that they do equal each other.)
We’ve found this to be because a small number of VolumetricLightmap tasks are being handled in FLightmassProcessor::SwarmCallback()'s JOB_TASK_STATE_COMPLETE_SUCCESS handler more than once. (Of the 125,000 tasks issued, it’s usually a few hundred that double report -- I haven’t seen more than 900 double report in a run.)
Adding a TSet<FGuid> to track which of these has reported in (and protecting it with a mutex) and not counting duplicate reports has fixed the issue for us. (We have not dug into why the double-reports are happening.)
ADDENDUM: I should have also pointed out that our Swarm workers are accessed remotely, over VPN, so we may have more packet-loss and latency than on-LAN setups.
Steps to Reproduce
Bake static lighting in a world partition map with r.AllowStaticLightingInWorldPartitionMaps
`==== //depot/Unreal/Engine/Source/Editor/UnrealEd/Private/Lightmass/Lightmass.cpp#3 (text) ====
[Content removed]18 @@
}
else if (Processor->Exporter->VolumetricLightmapTaskGuids.Contains(TaskStateMessage->TaskGuid))
{
- //~ — EDIT BEGIN — : pchaintreuil - 2025/04/15 - Don’t double count Lightmap Guids that report complete more than once.
- {
- UE::TScopeLock const ScopeLock (Processor->Exporter->VolumetricLightmapTaskGuidsOutstandingLock);
- if (Processor->Exporter->VolumetricLightmapTaskGuidsOutstanding.Remove(TaskStateMessage->TaskGuid) < 1)
- {
- // If a GUID is reporting complete again, don’t count it.
- break;
- }
- }
- //~ — EDIT END —/
TList* NewElement = new TList(TaskStateMessage->TaskGuid, NULL);
Processor->CompletedVolumetricLightmapTasks.AddElement( NewElement );
FPlatformAtomics::InterlockedIncrement( &Processor->NumCompletedTasks );
[Content removed]13 @@
{
Exporter->VolumetricLightmapTaskGuids.Add(FGuid::NewGuid(), VolumetricLightmapTaskIndex);
}
+
- //~ — EDIT BEGIN — : pchaintreuil - 2025/04/15 - Don’t double count Lightmap Guids that report complete more than once.
- {
- UE::TScopeLock const ScopeLock (Exporter->VolumetricLightmapTaskGuidsOutstandingLock);
- Exporter->VolumetricLightmapTaskGuids.GetKeys(Exporter->VolumetricLightmapTaskGuidsOutstanding);
- }
- //~ — EDIT END —/
}
TArray<ULevel*> LevelsToRestore;
==== //depot/Unreal/Engine/Source/Editor/UnrealEd/Private/Lightmass/Lightmass.h#2 (text) ====
[Content removed]13 @@
TMap<FGuid, int32> VolumetricLightmapTaskGuids;
- //~ — EDIT BEGIN — : pchaintreuil - 2025/04/15 - Don’t double count Lightmap Guids that report complete more than once.
- /** Volumetric Lightmap tasks that haven’t yet reported as completed. */
- TSet VolumetricLightmapTaskGuidsOutstanding;
- /** Mutex for VolumetricLightmapTaskGuidsOutstanding */
- FCriticalSection VolumetricLightmapTaskGuidsOutstandingLock;
- //~ — EDIT END —/
TArray<AVolumetricLightmapDensityVolume*> VolumetricLightmapDensityVolumes;
private:`
Hi Phillipe,
Thanks for reaching out about this to proactively contribute to the engine. Can you please submit this change as a pull request via Github and send me the link? That way, it will be easier for the team in charge of Lightmass to review the changes, and I will see about possibly expediting the change if needed.
Cheers,
Tim
Thanks for creating the PR. I made the dev team aware of this proposed change, and they will respond to you in a timely manner. If you have any further questions, please let me know.