Use after free due to realloc when FPakPrecacher::StartBlockTask run concurrently

Development Platform & Builds
, ,
1

We saw this exiting a WinGDK build while running under Address Sanitizer. The issue appears to be FPakPrecacher::AddNewBlock running on two threads at once. One of them allocates a new FCacheBlock, then starts using it:

TIntervalTreeIndex NewIndex = CacheBlockAllocator.Alloc();
FCacheBlock& Block = CacheBlockAllocator.Get(NewIndex);

At some point, another thread allocates its own new block. If this causes a reallocation, the block from the first thread gets moved and its original address is freed. The first thread then attempts to access this freed memory through its existing pointer.

One way to address this particular case might be to create and initialize Block on the stack, then allocate NewIndex and copy it in, and keep using the stack version for the rest of the function. But there are a number of other places in FPakPrecacher that also access blocks out of CacheBlockAllocator and could potentially go awry if memory got reallocated under them.

I don’t understand this whole area well enough to know what can run concurrently, so I wanted to see what Epic thinks. As we’ve only caught this on app exit it doesn’t seem critical, but I’m concerned it could theoretically happen at other times.

Note that we do have some engine customizations, but I didn’t see anything related to this. But if AddNewBlock isn’t supposed to run concurrently and this is something we’ve changed somewhere, that’s good to know too.

[Attachment Removed]

2

Hi,

Thanks for reporting. We are tracking this issue here UE-369226. Unfortunately this will most likely

be a low priority. The PAK system is only really used

for .ini files and other non-asset data if I/O store has been enabled on your project.

I know we’ve had issues with something similar before but we do lock around the block allocations.

/Per

[Attachment Removed]

3

It should be safe. I don’t see any relevant changes around the precacher block allocation UE 5.5 vs Fortnite main branch. I also haven’t seen anything in

our internal crash reports indicating that this is a problem.

Do you have callstack that you can provide?

/Per

[Attachment Removed]

4

How big is it? Can you try and email to [Content removed]

[Attachment Removed]

5

Update: We managed to find and fix the issue;

The precacher uses TIntervalTreeAllocator, a container with a TArray of requests blocks to process. The TArray reserves 4 entries, by default.

The processing is recursive and works until both of the following conditions are true;

- The recursive processing requests more than 4 blocks.

- The internal realloc of the TArray, moves the block array in memory. (invalidating all currently held block addresses)

This manifests as invalid memory access/corruption.

Our local fix simply replaces the request block TArray with TChunkArray, which is address stable on expansion. (which is the use case)

[Attachment Removed]

6

Thanks for the update. Sounds like an easy fix and I’ve added this to the tracking Jira.

[Attachment Removed]

7

Thanks. We do use IoStore, so this must coming from some piece of data still using the pak system.

Do you think this is safe for us to ignore? I’m only seeing it on exit, but it is consistent in ASAN. I’m not aware of any actual crashes we’ve seen known to be tied to it.

[Attachment Removed]

8

I’ve attached the ASAN output. It looks like this happens when the FShaderPipelineCache shuts down and tries to save out its data.

We recently added a block at startup to wait for PSOs to be ready before you can interact, so that will probably prevent this scenario, but I need to test that. We are wondering if it’s possible to hit it if you alt+F4 during that wait, and if that could corrupt anything.

[Attachment Removed]

9

It’s saying the file I uploaded isn’t available. (I’d also attached it to the original post and I guess it didn’t come through there either.) Is there any other way I can share it? It’s too big for a reply.

[Attachment Removed]

10

Here’s a callstack just copied out of Visual Studio of where ASAN catches this:

>	GameName-WinGDK-Test.exe!FPakPrecacher::StartBlockTask(FPakPrecacher::FCacheBlock & Block) Line 1992	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::AddNewBlock() Line 1829	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::StartNextRequest() Line 2117	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::NewRequestsToLowerComplete(bool bWasCanceled, IAsyncReadRequest * Request, int Index) Line 2220	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::StartBlockTask::__l2::<lambda>(bool bWasCanceled, IAsyncReadRequest * Request) Line 1985	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!Invoke(FPakPrecacher::StartBlockTask::__l2::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &) &) Line 47	C++
 	GameName-WinGDK-Test.exe!UE::Core::Private::Function::TFunctionRefCaller<`FPakPrecacher::StartBlockTask'::`2'::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &),void,bool,IAsyncReadRequest *>::Call(void * Obj, bool & <Params_0>, IAsyncReadRequest * & <Params_1>) Line 322	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<0>,void __cdecl(bool,IAsyncReadRequest *)>::operator()(bool <Params_0>, IAsyncReadRequest * <Params_1>) Line 471	C++
 	GameName-WinGDK-Test.exe!IAsyncReadRequest::SetDataComplete() Line 214	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!IAsyncReadRequest::SetComplete() Line 225	C++
 	GameName-WinGDK-Test.exe!FGenericReadRequest::PerformRequest() Line 543	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!FGenericReadRequestWorker::DoWork() Line 273	C++
 	GameName-WinGDK-Test.exe!FAsyncTask<FGenericReadRequestWorker>::DoTaskWork() Line 656	C++
 	GameName-WinGDK-Test.exe!FAsyncTaskBase::DoWork() Line 290	C++
 	GameName-WinGDK-Test.exe!FAsyncTaskBase::Abandon() Line 331	C++
 	GameName-WinGDK-Test.exe!FQueuedThreadPoolBase::AddQueuedWork(IQueuedWork * InQueuedWork, EQueuedWorkPriority InQueuedWorkPriority) Line 1402	C++
 	GameName-WinGDK-Test.exe!FAsyncTaskBase::Start(bool bForceSynchronous, FQueuedThreadPool * InQueuedPool, EQueuedWorkPriority InQueuedWorkPriority, EQueuedWorkFlags InQueuedWorkFlags, __int64 InRequiredMemory, const wchar_t * InDebugName) Line 271	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!FAsyncTaskBase::StartBackgroundTask(FQueuedThreadPool *) Line 421	C++
 	GameName-WinGDK-Test.exe!FGenericBaseRequest::Start() Line 83	C++
 	GameName-WinGDK-Test.exe!FGenericReadRequest::FGenericReadRequest(FGenericAsyncReadFileHandle * InOwner, IPlatformFile * InLowerLevel, const wchar_t * InFilename, TFunction<void __cdecl(bool,IAsyncReadRequest *)> * CompleteCallback, unsigned char * UserSuppliedMemory, __int64 InOffset, __int64 InBytesToRead, EAsyncIOPriorityAndFlags InPriorityAndFlags) Line 226	C++
 	GameName-WinGDK-Test.exe!FGenericAsyncReadFileHandle::ReadRequest(__int64 Offset, __int64 BytesToRead, EAsyncIOPriorityAndFlags PriorityAndFlags, TFunction<void __cdecl(bool,IAsyncReadRequest *)> * CompleteCallback, unsigned char * UserSuppliedMemory) Line 371	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::StartBlockTask(FPakPrecacher::FCacheBlock & Block) Line 1987	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::AddNewBlock() Line 1829	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!FPakPrecacher::StartNextRequest() Line 2114	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::AddRequest(FPakPrecacher::FPakInRequest & Request, unsigned int NewIndex) Line 1173	C++
 	GameName-WinGDK-Test.exe!FPakPrecacher::QueueRequest(IPakRequestor * Owner, FPakFile * InActualPakFile, FName File, __int64 PakFileSize, __int64 Offset, __int64 Size, EAsyncIOPriorityAndFlags PriorityAndFlags) Line 2260	C++
 	GameName-WinGDK-Test.exe!FPakReadRequest::FPakReadRequest(FPakFile * InActualPakFile, FName InPakFile, __int64 PakFileSize, TFunction<void __cdecl(bool,IAsyncReadRequest *)> * CompleteCallback, __int64 InOffset, __int64 InBytesToRead, EAsyncIOPriorityAndFlags InPriorityAndFlags, unsigned char * UserSuppliedMemory, bool bInInternalRequest, FCachedAsyncBlock * InBlockPtr) Line 2743	C++
 	GameName-WinGDK-Test.exe!FPakAsyncReadFileHandle::ReadRequest(__int64 Offset, __int64 BytesToRead, EAsyncIOPriorityAndFlags PriorityAndFlags, TFunction<void __cdecl(bool,IAsyncReadRequest *)> * CompleteCallback, unsigned char * UserSuppliedMemory) Line 3366	C++
 	GameName-WinGDK-Test.exe!`FPipelineCacheFile::SavePipelineFileCache'::`21'::FMemoryReaderAndMemory::FMemoryReaderAndMemory(FPipelineCacheFile * PipelineFile) Line 2457	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!MakeUnique(FPipelineCacheFile * &) Line 798	C++
 	GameName-WinGDK-Test.exe!FPipelineCacheFile::SavePipelineFileCache::__l21::<lambda>(const FGuid & guid) Line 2487	C++
 	GameName-WinGDK-Test.exe!FPipelineCacheFile::SavePipelineFileCache(FPipelineFileCacheManager::SaveMode Mode, const TMap<unsigned int,FPipelineStateStats *,FDefaultSetAllocator,TDefaultMapHashableKeyFuncs<unsigned int,FPipelineStateStats *,0>> & Stats, TSet<FPipelineCacheFileFormatPSO,DefaultKeyFuncs<FPipelineCacheFileFormatPSO,0>,FDefaultSetAllocator> & NewEntries, FPipelineFileCacheManager::PSOOrder Order, TMap<unsigned int,FPSOUsageData,FDefaultSetAllocator,TDefaultMapHashableKeyFuncs<unsigned int,FPSOUsageData,0>> & NewPSOUsage) Line 2688	C++
 	GameName-WinGDK-Test.exe!FPipelineFileCacheManager::SavePipelineFileCache(FPipelineFileCacheManager::SaveMode Mode) Line 3454	C++
 	GameName-WinGDK-Test.exe!FShaderPipelineCache::SavePipelineFileCache(FPipelineFileCacheManager::SaveMode Mode) Line 958	C++
 	GameName-WinGDK-Test.exe!FShaderPipelineCache::CloseUserPipelineFileCache() Line 980	C++
 	GameName-WinGDK-Test.exe!FShaderPipelineCache::~FShaderPipelineCache() Line 1749	C++
 	GameName-WinGDK-Test.exe!FShaderPipelineCache::`scalar deleting destructor'(unsigned int)	C++
 	GameName-WinGDK-Test.exe!FShaderPipelineCache::Shutdown() Line 782	C++
 	GameName-WinGDK-Test.exe!FEngineLoop::Exit() Line 4983	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!EngineExit() Line 74	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!GuardedMain::__l2::EngineLoopCleanupGuard::{dtor}() Line 132	C++
 	GameName-WinGDK-Test.exe!GuardedMain(const wchar_t * CmdLine) Line 212	C++
 	GameName-WinGDK-Test.exe!LaunchWindowsStartup(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow, const wchar_t * CmdLine) Line 288	C++
 	GameName-WinGDK-Test.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * pCmdLine, int nCmdShow) Line 369	C++
 	[Inline Frame] GameName-WinGDK-Test.exe!invoke_main() Line 102	C++
 	GameName-WinGDK-Test.exe!__scrt_common_main_seh() Line 288	C++
 	kernel32.dll!00007ffb82b4e8d7()	Unknown
 	ntdll.dll!00007ffb8466c53c()	Unknown

I’ve confirmed that we still see this even if waiting for shader compilation to finish before exiting.

[Attachment Removed]

11

Thanks, I just sent it over.

[Attachment Removed]