ParallelFor memory safety issue caught by Stomp2Malloc in 5.6

Development Platform & Builds
, ,
1

Hi Epic. We’ve recently switched to 5.6 for certain projects, and our stomp2malloc test is catching an issue with the ParallelFor.

​

Callstack 1 (CI):

Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION writing address 0x0000021532050000
Error: 
Error: [Callstack] 0x00007ffdf7416ae1 UnrealEditor-Projects.dll!TLinearArrayAllocatorBase<FTaskGraphBlockAllocationTag,0>::ForElementType<`ParallelForImpl::ParallelForInternal<TFunctionRef<void __cdecl(int)>,`ParallelFor'::`2'::<lambda_1>,std::nullptr_t>'::`2'::FTracedTask>::~ForElementType<`ParallelForImpl::ParallelForInternal<TFunctionRef<void __cdecl(int)>,`ParallelFor'::`2'::<lambda_1>,std::nullptr_t>'::`2'::FTracedTask>() [{ROOT}\Main.Build\UE5\Engine\Source\Runtime\Core\Public\Experimental\ConcurrentLinearAllocator.h:626]
Error: [Callstack] 0x00007ffdf742237a UnrealEditor-Projects.dll!`ParallelForImpl::ParallelForInternal<TFunctionRef<void __cdecl(int)>,`ParallelFor'::`2'::<lambda_1>,std::nullptr_t>'::`2'::FParallelForData::`scalar deleting destructor'() []
Error: [Callstack] 0x00007ffdf740e2e9 UnrealEditor-Projects.dll!ParallelForImpl::ParallelForInternal<TFunctionRef<void __cdecl(int)>,`ParallelFor'::`2'::<lambda_1>,std::nullptr_t>() [{ROOT}\Main.Build\UE5\Engine\Source\Runtime\Core\Public\Async\ParallelFor.h:471]
Error: [Callstack] 0x00007ffdf7450515 UnrealEditor-Projects.dll!FPluginManager::ReadPluginsInDirectory() [{ROOT}\Main.Build\UE5\Engine\Source\Runtime\Projects\Private\PluginManager.cpp:1054]
Error: [Callstack] 0x00007ffdf744ce6f UnrealEditor-Projects.dll!FPluginManager::ReadAllPlugins() [{ROOT}\Main.Build\UE5\Engine\Source\Runtime\Projects\Private\PluginManager.cpp:807]
Error: [Callstack] 0x00007ffdf742f9f1 UnrealEditor-Projects.dll!FPluginManager::DiscoverAllPlugins() [{ROOT}\Main.Build\UE5\Engine\Source\Runtime\Projects\Private\PluginManager.cpp:758]
Error: [Callstack] 0x00007ffdf741521e UnrealEditor-Projects.dll!FPluginManager::FPluginManager() [{ROOT}\Main.Build\UE5\Engine\Source\Runtime\Projects\Private\PluginManager.cpp:538]
...

​

Callstack 2 (with VS):

....
[Inline Frame] UnrealEditor-Core.dll!TRefCountPtr<`ParallelForImpl::ParallelForInternal<`IPlatformFile::IterateDirectoryRecursively'::`4'::void <lambda>(void),`ParallelForWithTaskContext<TArray<FString,TSizedDefaultAllocator<32>>,TSizedDefaultAllocator<32>,`IPlatformFile::IterateDirectoryRecursively'::`4'::void <lambda>(void)>'::`5'::void <lambda>(void),TArray<FString,TSizedDefaultAllocator<32>>>'::`2'::FParallelForData>::{dtor}() Line 505	C++
[Inline Frame] UnrealEditor-Core.dll!ParallelForImpl::ParallelForInternal::__l2::FParallelForData::{dtor}() Line 117	C++
...
[Inline Frame] UnrealEditor-Core.dll!ParallelForImpl::ParallelForInternal::__l2::FParallelExecutor::{dtor}() Line 117	C++
...
UnrealEditor-Core.dll!LowLevelTasks::FScheduler::WorkerLoop(LowLevelTasks::Private::FWaitEvent * WorkerEvent, LowLevelTasks::Private::TLocalQueueRegistry<1024,1024>::TLocalQueue * WorkerLocalQueue, unsigned int WaitCycles, bool bPermitBackgroundWork) Line 724	C++
...

​

(abbreviated callstacks due to the post char limit)

​

It seems it’s something to do with the destruction of the FParallelForData in both cases, but it’s not clear to me.

​

I was able to repro in an almost-vanilla version of UE 5.6. The issue does not repro with 5.5.

​

We would appreciate a fix, as this test is no longer able to enter any of our code.

​

Thanks.

2

Steps to Reproduce
Launch the editor with:

-stomp2malloc -MallocStomp2MaxSize=9999999

The first call to a ParallelFor should repro the issue.

3

Hi Kostas,

There is a known issue currently where stomp2malloc doesn’t support to return the proper alignment for the linear allocator to function correctly.

In the meantime, you can change this to false in the linear allocator to be able to work with stomp3malloc.

static constexpr bool SupportsFastPath = false;Hope this helps

Danny

4

Of course! someone is already tasked with fixing stomp2malloc to support this.

I can let you know once the fix is in.

Have a good day!

Danny

5

So that was fast :slight_smile:

It’s been fixed in 46421205 in UE5/Main

Let me know if any problems comes up with this patch applied.

Thanks

Danny

6

Thanks Danny. That fixed the issue.

I had to pass a new #define from our build system, to make this change conditional on running the S2M test:

static constexpr bool SupportsFastPath = ...
		&& !NT_STOMP2MALLOC_ENABLED;

It would be good to have an official solution one day. Maybe a standardised #define, plus a mention of it in the documentation of S2M, since S2M will no longer be just a runtime decision. Or, even better, a solution to the root cause inside the linear allocator, if possible. The latter would also avoid the huge recompilation that results from the extra #define.

7

Thanks! :slight_smile: I see there is another change in between 5.6 and this new one today, so I had best not risk it. We’ll undo my hack come 5.7+, at which point I’ll start a fresh post if there are any issues.