Packaging fails with substepping enabled

bent.gross · August 28, 2024, 8:10am

Hi,
as long as substepping is enabled in the project settings packaging the project always fails with a fatal error inside FScopedSceneLock_Chaos!
Are there any information why substepping would cause a fatal error while packaging?

PackagingResults: Error: begin: stack for UAT
PackagingResults: Error: === Critical error: ===
PackagingResults: Error: Fatal error!
PackagingResults: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0xffffffffffffffff
PackagingResults: Error: [Callstack] 0x00007ffb94d6e481 UnrealEditor-Engine.dll!FScopedSceneLock_Chaos::FScopedSceneLock_Chaos() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\PhysicsEngine\Experimental\ChaosScopedSceneLock.cpp:108]
PackagingResults: Error: [Callstack] 0x00007ffb94e8030c UnrealEditor-Engine.dll!FPhysicsObjectExternalInterface::LockWrite() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\PhysicsEngine\PhysicsObjectExternalInterface.cpp:29]
PackagingResults: Error: [Callstack] 0x00007ffb93dc21b4 UnrealEditor-Engine.dll!UPrimitiveComponent::OnDestroyPhysicsState() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\PrimitiveComponent.cpp:968]
PackagingResults: Error: [Callstack] 0x00007ffb93c3df9f UnrealEditor-Engine.dll!UActorComponent::DestroyPhysicsState() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\ActorComponent.cpp:1737]
PackagingResults: Error: [Callstack] 0x00007ffb93c4c824 UnrealEditor-Engine.dll!UActorComponent::ExecuteUnregisterEvents() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\ActorComponent.cpp:1778]
PackagingResults: Error: [Callstack] 0x00007ffb93c1e9d5 UnrealEditor-Engine.dll!UActorComponent::BeginDestroy() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\ActorComponent.cpp:768]
PackagingResults: Error: [Callstack] 0x00007ffb93d7f8d5 UnrealEditor-Engine.dll!UPrimitiveComponent::BeginDestroy() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\PrimitiveComponent.cpp:1553]
PackagingResults: Error: [Callstack] 0x00007ffbc1dc1a46 UnrealEditor-CoreUObject.dll!UObject::ConditionalBeginDestroy() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\Obj.cpp:1214]
PackagingResults: Error: [Callstack] 0x00007ffbc1cf970c UnrealEditor-CoreUObject.dll!UnhashUnreachableObjects() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:5949]
PackagingResults: Error: [Callstack] 0x00007ffbc1c01203 UnrealEditor-CoreUObject.dll!UE::GC::PostCollectGarbageImpl<1>() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:5575]
PackagingResults: Error: [Callstack] 0x00007ffbc1c8ac2d UnrealEditor-CoreUObject.dll!UE::GC::FReachabilityAnalysisState::PerformReachabilityAnalysisAndConditionallyPurgeGarbage() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:5771]
PackagingResults: Error: [Callstack] 0x00007ffbc1c350d9 UnrealEditor-CoreUObject.dll!CollectGarbage() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:6007]
PackagingResults: Error: [Callstack] 0x00007ffba596ceed UnrealEditor-UnrealEd.dll!UCookCommandlet::ConditionalCollectGarbage() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:699]
PackagingResults: Error: [Callstack] 0x00007ffba59adb4d UnrealEditor-UnrealEd.dll!UCookCommandlet::RunCookByTheBookCook() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:551]
PackagingResults: Error: [Callstack] 0x00007ffba596f4ed UnrealEditor-UnrealEd.dll!UCookCommandlet::CookByTheBook() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:508]
PackagingResults: Error: [Callstack] 0x00007ffba598aca0 UnrealEditor-UnrealEd.dll!UCookCommandlet::Main() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:249]
PackagingResults: Error: [Callstack] 0x00007ff7e5f980dd UnrealEditor-Cmd.exe!FEngineLoop::PreInitPostStartupScreen() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:4195]
PackagingResults: Error: [Callstack] 0x00007ff7e5f8e146 UnrealEditor-Cmd.exe!GuardedMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Launch.cpp:136]
PackagingResults: Error: [Callstack] 0x00007ff7e5f8e42a UnrealEditor-Cmd.exe!GuardedMainWrapper() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:118]
PackagingResults: Error: [Callstack] 0x00007ff7e5f918a4 UnrealEditor-Cmd.exe!LaunchWindowsStartup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:258]
PackagingResults: Error: [Callstack] 0x00007ff7e5fa70c4 UnrealEditor-Cmd.exe!WinMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:298]
PackagingResults: Error: [Callstack] 0x00007ff7e5fa96e6 UnrealEditor-Cmd.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
PackagingResults: Error: [Callstack] 0x00007ffc2fc17374 KERNEL32.DLL!UnknownFunction []
PackagingResults: Error: end: stack for UAT

The build then failes with ExitCode=25 (Error_UnknownCookFailure)

But the cooking process semmed to be successfully finished before:

 Packaging (Windows): LogInit: Display: Success - 0 error(s), 175 warning(s)

The warning are almost entierly Input Actions which some demo actors are using.

This whole scenario with the build failing because of phyiscs substepping is quite strange for me.
As the fatal error occurs inside BeginDestroy, why do actors get destroyed while packaging?

Best regards

bent.gross · August 30, 2024, 3:12pm

Update:
I did find out that substepping seemed not to be the only problem but it made it worse!
Certain maps in the project do build without substepping enabled while others still create the mentioned fatal error.
BUT with substepping enabled it does not matter which maps I include in the build it will fail with the mentioned fatal error.

So, the question is still to be answered, how does this error be even possible while cooking/packaging? Why do certain phyiscs be updated at cook time?

best regards

nixonkuw · November 6, 2024, 2:09am

Wanted to bump this, seeing as well intermittently and frequently on 5.4.3.

A full workspace clean seems to get around this but this is unfeasible, and the issue reoccurs after some time.
Unsure of the initial cause since it didn’t seem to occur determinately.
Output shown below, seems to mention a successful cook and this occurs immediately after.

LogInit: Display: Success - 0 error(s), 314 warning(s)
LogInit: Display:
Execution of commandlet took:  2m 11s (131.95 seconds)
LogWindows: Error: begin: stack for UAT
LogWindows: Error: === Critical error: ===
LogWindows: Error:
LogWindows: Error: Fatal error!
LogWindows: Error:
LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION writing address 0x0000000b00000007
LogWindows: Error:
LogWindows: Error: [Callstack] 0x00007ffcca7bfaad ntdll.dll!UnknownFunction []
LogWindows: Error: [Callstack] 0x00007ffc91516118 UnrealEditor-Engine.dll!FScopedSceneLock_Chaos::LockScene() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\PhysicsEngine\Experimental\ChaosScopedSceneLock.cpp:156]
LogWindows: Error: [Callstack] 0x00007ffc914ac986 UnrealEditor-Engine.dll!FScopedSceneLock_Chaos::FScopedSceneLock_Chaos() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\PhysicsEngine\Experimental\ChaosScopedSceneLock.cpp:112]
LogWindows: Error: [Callstack] 0x00007ffc915cb63c UnrealEditor-Engine.dll!FPhysicsObjectExternalInterface::LockWrite() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\PhysicsEngine\PhysicsObjectExternalInterface.cpp:29]
LogWindows: Error: [Callstack] 0x00007ffc904f0774 UnrealEditor-Engine.dll!UPrimitiveComponent::OnDestroyPhysicsState() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Components\PrimitiveComponent.cpp:968]
LogWindows: Error: [Callstack] 0x00007ffc9065115e UnrealEditor-Engine.dll!UStaticMeshComponent::OnDestroyPhysicsState() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Components\StaticMeshComponent.cpp:827]
LogWindows: Error: [Callstack] 0x00007ffc9035fd08 UnrealEditor-Engine.dll!UActorComponent::DestroyPhysicsState() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Components\ActorComponent.cpp:1737]
LogWindows: Error: [Callstack] 0x00007ffc9036e764 UnrealEditor-Engine.dll!UActorComponent::ExecuteUnregisterEvents() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Components\ActorComponent.cpp:1778]
LogWindows: Error: [Callstack] 0x00007ffc903f3a40 UnrealEditor-Engine.dll!UActorComponent::UnregisterComponent() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Components\ActorComponent.cpp:1502]
LogWindows: Error: [Callstack] 0x00007ffc8fbed552 UnrealEditor-Engine.dll!AActor::UnregisterAllComponents() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Actor.cpp:5338]
LogWindows: Error: [Callstack] 0x00007ffc8fb68487 UnrealEditor-Engine.dll!AActor::BeginDestroy() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Engine\Private\Actor.cpp:745]
LogWindows: Error: [Callstack] 0x00007ffc945aae16 UnrealEditor-CoreUObject.dll!UObject::ConditionalBeginDestroy() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\CoreUObject\Private\UObject\Obj.cpp:1214]
LogWindows: Error: [Callstack] 0x00007ffc944dcce0 UnrealEditor-CoreUObject.dll!UnhashUnreachableObjects() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:5949]
LogWindows: Error: [Callstack] 0x00007ffc944663f7 UnrealEditor-CoreUObject.dll!IncrementalPurgeGarbage() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:4572]
LogWindows: Error: [Callstack] 0x00007ffc945ecd82 UnrealEditor-CoreUObject.dll!StaticExit() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\CoreUObject\Private\UObject\Obj.cpp:5439]
LogWindows: Error: [Callstack] 0x00007ffc94436c37 UnrealEditor-CoreUObject.dll!TBaseStaticDelegateInstance<void __cdecl(void),FDefaultDelegateUserPolicy>::ExecuteIfSafe() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h:777]
LogWindows: Error: [Callstack] 0x00007ff733af20c1 UnrealEditor-Cmd.exe!TMulticastDelegate<void __cdecl(void),FDefaultDelegateUserPolicy>::Broadcast() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Core\Public\Delegates\DelegateSignatureImpl.inl:956]
LogWindows: Error: [Callstack] 0x00007ff733af1792 UnrealEditor-Cmd.exe!FEngineLoop::AppPreExit() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:6893]
LogWindows: Error: [Callstack] 0x00007ff733afccc1 UnrealEditor-Cmd.exe!FEngineLoop::Exit() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:5159]
LogWindows: Error: [Callstack] 0x00007ff733afee5b UnrealEditor-Cmd.exe!GuardedMain() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Launch\Private\Launch.cpp:194]
LogWindows: Error: [Callstack] 0x00007ff733afeeba UnrealEditor-Cmd.exe!GuardedMainWrapper() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:118]
LogWindows: Error: [Callstack] 0x00007ff733b02384 UnrealEditor-Cmd.exe!LaunchWindowsStartup() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:258]
LogWindows: Error: [Callstack] 0x00007ff733b17e04 UnrealEditor-Cmd.exe!WinMain() [C:\Horde\ABC\Sync\Unreal\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:298]
LogWindows: Error: [Callstack] 0x00007ff733b1a426 UnrealEditor-Cmd.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
LogWindows: Error: [Callstack] 0x00007ffcc9eb7374 KERNEL32.DLL!UnknownFunction []
LogWindows: Error:
LogWindows: Error: end: stack for UAT
Took 146.27s to run UnrealEditor-Cmd.exe, ExitCode=3
Copying crash data to C:\Horde\ABC\Sync\Unreal\Engine\Programs\AutomationTool\Saved\Logs\BuildCookRun\Crashes\UECC-Windows-...
Cook failed.
(see C:\Horde\ABC\Sync\Unreal\Engine\Programs\AutomationTool\Saved\Logs\BuildCookRun\Log.txt for full exception trace)
AutomationTool executed for 0h 2m 28s
AutomationTool exiting with ExitCode=25 (Error_UnknownCookFailure)

CrashContext:

		<SourceContext>  141       
  142       void FScopedSceneLock_Chaos::LockScene()
  143       {
  144       	if (!Solver)
  145       	{
  146       		return;
  147       	}
  148       
  149       	switch (LockType)
  150       	{
  151       	case EPhysicsInterfaceScopedLockType::Read:
  152       		Solver-&gt;GetExternalDataLock_External().ReadLock();
  153       		break;
  154       	case EPhysicsInterfaceScopedLockType::Write:
  155 ***** 		Solver-&gt;GetExternalDataLock_External().WriteLock();
  156       		break;
  157       	}
  158       
  159       	bHasLock = true;
  160       }
  161       
  162       void FScopedSceneLock_Chaos::UnlockScene()
  163       {
  164       	if (!Solver)
  165       	{
  166       		return;
  167       	}
  168       
  169       	switch (LockType)
  170       	{</SourceContext>

geoffstacey_work · November 8, 2024, 2:27am

Hi nixonkuw,
This has been raised elsewhere with a back link to this issue. We believe this is being caused by multithreaded code destructing the physics solvers during teardown after the build.
I’ll need some debugging info to confirm this, which I have asked for elsewhere. Happy to include you if you are familiar with debugging and Visual Studio!
Best
Geoff Stacey
Developer Relations
Epic Games