Lightmass static lightning timed-out, asserts mismatch in task counting

Development Rendering
, , , , , ,
1

I’m building lightning in UE 4.23.0 on Ubuntu 16.04 and, following the logs, when static lightning needs to be built it gets timed out. Later, in the logs, an error LogInit: Error: Timed out waiting for the recipient (TimeWaitingSec = 60.015285) and afterwards an assert on Volumentric Lightmap:

LogCore: Error: appError called: Assertion failed: TaskDataArray.Num() == Exporter->VolumetricLightmapTaskGuids.Num() [File:/opt/UnrealEngine/Engine/Source/Editor/UnrealEd/Private/Lightmass/ImportVolumetricLightmap.cpp] [Line: 956]

Import Volumetric Lightmap failed: Expected 3047495776 tasks, only found 3073429500

Any clue on what is happening and how to fix it is very welcome. Mind that I have loopback address set up correctly as mentioned here and I had no problems doing it in UE version 4.22.3.

1 Like
2

Note: The error persists even after doing a clean build (repository freshly cloned directly from the 4.23.0-release tag).

3

Just want to report that later releases fixed it, but I have still have no idea what fixed it.

4

I ran into the same issue, and worked around the problem.

Some background:

I self-built 4.23.1 to test out the linux pixel streaming. Since 4.23.1 is the only version that supports linux pixel streaming now, I can’t upgrade to the latest version.

I purchased a finished game level from Epic’s marketplace, but after a few modifications, I saw the same error when trying to rebuild lighting.

I googled, the top answer is enabling multicast for the loopback network interface. Others suggested disabling the ufw firewall.

Looking at the message, it seems to be a communication issue between the swarm server / or the lightmass process. (I’m very new to unreal, so not sure if I’m interpreting it correctly).

Later I learned that Unreal doesn’t support swarm, which is a rendering server, on linux. Instead, we need to use the local version of swarm, which should be transparent to user.

Since all internet solutions didn’t work, I debugged the C++ code, here is the problematic part:

int32 FSwarmInterfaceLocalImpl::SendMessage( const FMessage& Message )
{

#if USE_LOCAL_SWARM_INTERFACE
...
	while (bIsConnected && !Recepient.IsValid())
	{

		MessageEndpoint->Publish(new FSwarmPingMessage(), EMessageScope::Network);
...
	if (TimeWaitingSec >= kMaxTimeToWaitSec)
		{
			UE_LOG(LogInit, Error, TEXT("Timed out waiting for the recipient (TimeWaitingSec = %f)"), TimeWaitingSec);
			return -1;
		}
		
	}

...

Basically the code tries to issue a ping message to another process and expect a pong message, as a way to verify an interprocess communication.

But it times out. I then found the pong message handler:

void FSwarmInterfaceLocalImpl::HandlePongMessage( const FSwarmPongMessage& Message, const TSharedRef<IMessageContext, ESPMode::ThreadSafe>& Context )
{
	if (!Recepient.IsValid() && Message.bIsEditor != bIsEditor && Message.ComputerName == FPlatformProcess::ComputerName())
	{
		Recepient = Context->GetSender();
	}
}

This handler is called as soon as the ping message is sent. This means that the communication is good. However, the if condition fails and results in the timeout.

The failed condition is actually “Message.bIsEditor != bIsEditor” , this seems to be a condition to verify that the message is indeed heard from another process, not a loopback message.

Somehow, both Message.bIsEditor and bIsEditor are zeros. With some further debugging, I feel that bIsEditor is correct. But Message.bIsEditor is wrong.

But anyway, I hacked it to Message.bIsEditor == bIsEditor, my lightmaps seem to build now. But it has been 8 hours and not finished. Who knows if I will see new problems.

2 Likes
5

As someone who ran into this same issue on macOS and similarly had to debug the code to see what was going on, I just wanted to expand on your comment as it mirrored what I was seeing.

After the UnrealEditor process spawns the UnrealLightmass subprocess, both processes subscribe to all multicast ping messages (regardless of sender). This means that a ping-emitting process will receive and handle its own ping messages.

In the unhappy case we observed, both the editor and lightmass processes only receive their own ping message. This is why Message.bIsEditor is equal to bIsEditor in your observed case (both variables would be true within the editor process and false within the lightmass process).

In the happy case when everything is working, both processes will be capable of receiving both their own ping messages as well as each other’s ping messages. In practice, this allows the UnrealLightmass process to deliver a ping to the UnrealEditor process and for the editor to respond with a pong message. The UnrealLightmass process uses the sender message address from the pong message as the recipient for all subsequent messages to relay lighting results back to the editor.

As this interprocess communication relies on multicast (with a group address of 230.0.0.1:6666, as of today), networking software like firewalls and VPNs may contribute to a breakdown in communication between these processes resulting in light build failures. In my case, my VPN was to blame. Once I disabled the VPN (as it lacks granular settings to permit the traffic), my light builds began to succeed.

6

I’m using the ResavePackage commandlet to build lighting on my builder (Apple Silicon, Unreal 5.4), but I faced a similar issue. UnrealEditor-Cmd was not communicating with the UnrealLightmass subprocess, even though I had disabled all firewalls. I realized that only UnrealLightmass was listening to port 6666. After some debugging, I discovered that UnrealEditor-Cmd was not starting the UdpMessaging system and hence didn’t listen to the port. As a workaround, I added -messaging to the build command forcing the UnrealEditor-Cmd to start the UdpMessagin system, and I finally got it running.

Here is my final command to automate building lighting:

[BUILDER_WORKSPACE]/UnrealEngine/Engine/Binaries/Mac/UnrealEditor-Cmd "[BUILDER_WORKSPACE]/Project/Project.uproject" -run=ResavePackages -buildtexturestreaming -buildlighting -messaging -quality=Preview -MapsOnly -ProjectOnly -AllowCommandletRendering -SkipSkinVerify -Map=MainMap

1 Like