It reproduces randomly during build, with roughly a one-in-a-thousand probability. The crash stack appears to be related to multithreaded access to m_casLookup of StorageImpl.
The crash dump file is in the attachment.
重现步骤
It reproduces randomly during build, with roughly a one-in-a-thousand probability. The crash stack appears to be related to multithreaded access to m_casLookup of StorageImpl.
thanks for the report.. I will take a look and see if I can identify the problem.
Do you have any more information on this one. Does it happen early in the build? middle, or end?
hmmm, it would be great to get a dmp file that has more info (other thread’s callstacks).
We are running an insane amount of builds at Epic and I have not seen this issue. I looked through the code but it seems all access to that map is behind locks so my next guess would be double deletes/memory corruption. Can you check if all the dmp-files have the same callstack?
For reference, our cache server has served 150,000 clients the last two months (that is 2500 builds per day)… and our full clean builds are not even using the cache, only incremental/preflights..
So something is different between your farm and ours that brings this bug to the surface. It could ofc be that the bug was fixed.. you can test new binaries if you want, they should be fully backward compatible
I spent some time looking at those dumps and also the code and I can’t see how there can be a race on that map.. I think this is a memory overwrite or double delete or something like that. I remember finding a bug that I could imagine could cause memory issues a bunch of months ago or so.
You could test latest binaries in ue5-main if you want. The code has moved a lot since 5.6 (which I think was cut off for us in June or something like that) but it is supposed to backward compatible even though it has not been tested.
Worst case I think we need to implement a stomp allocator or something like that to find this issue
Sounds good. There are even more good fixes after 5.7... it is almost like we need to introduce a different release channel for UBA (since it does not depend on UE).
I’ve found this issue occurs randomly during code compilation, mostly in the middle of the build process.
We’ve integrated the code compilation into our code review workflow: every source code submission via P4 Swarm triggers a compilation. We have about 100 to 200 such submissions per day, and the compilation crash happens approximately once a week.
This issue has no reproducible pattern, but it does exist. We only have the dmp file containing the crash stack.
I’ve checked other dump files and identified two types of call stacks:
One crashed in StorageImpl::GetOrCreateCasEntry()
[Inline Frame] UbaHost.dll!uba::CasKey::operator==(const uba::CasKey &) Line 31 C++
[Inline Frame] UbaHost.dll!std::equal_to<uba::CasKey>::operator()(const uba::CasKey &) Line 393 C++
[Inline Frame] UbaHost.dll!std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>::operator()(const uba::CasKey &) Line 152 C++
[Inline Frame] UbaHost.dll!std::_Hash<std::_Umap_traits<uba::CasKey,uba::StorageImpl::CasEntry,std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>,uba::Allocator<std::pair<uba::CasKey const ,uba::StorageImpl::CasEntry>>,0>>::_Find_last(const uba::CasKey &) Line 1574 C++
[Inline Frame] UbaHost.dll!std::_Hash<std::_Umap_traits<uba::CasKey,uba::StorageImpl::CasEntry,std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>,uba::Allocator<std::pair<uba::CasKey const ,uba::StorageImpl::CasEntry>>,0>>::_Find(const uba::CasKey &) Line 1213 C++
[Inline Frame] UbaHost.dll!std::_Hash<std::_Umap_traits<uba::CasKey,uba::StorageImpl::CasEntry,std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>,uba::Allocator<std::pair<uba::CasKey const ,uba::StorageImpl::CasEntry>>,0>>::find(const uba::CasKey &) Line 1225 C++
UbaHost.dll!uba::StorageImpl::GetOrCreateCasEntry(const uba::CasKey & casKey) Line 927 C++
UbaHost.dll!uba::StorageImpl::AddCasFile(uba::StringKey fileNameKey, const wchar_t * fileName, const uba::CasKey & casKey, bool deferCreation, unsigned __int64 fileSize, unsigned __int64 lastWriteTime) Line 510 C++
[Inline Frame] UbaHost.dll!uba::StorageImpl::StoreCasFile::__l2::<lambda_1>::operator()() Line 1601 C++
[Inline Frame] UbaHost.dll!uba::ScopeGuard<`uba::StorageImpl::StoreCasFile'::`2'::<lambda_1>>::{dtor}() Line 176 C++
UbaHost.dll!uba::StorageImpl::StoreCasFile(uba::CasKey & out, const wchar_t * fileName, const uba::CasKey & casKeyOverride, bool deferCreation) Line 1706 C++
UbaHost.dll!uba::SessionServer::StoreCasFile(uba::CasKey & out, const uba::StringKey & fileNameKey, const wchar_t * fileName) Line 1873 C++
UbaHost.dll!uba::SessionServer::HandleGetFileFromServer(const uba::ConnectionInfo & workContext, const uba::WorkContext & reader, uba::BinaryReader & writer, uba::BinaryWriter &) Line 1038 C++
UbaHost.dll!uba::SessionServer::{ctor}::__l2::<lambda_2>::operator()(const uba::ConnectionInfo & connectionInfo, const uba::WorkContext & workContext, uba::MessageInfo & messageInfo, uba::BinaryReader & reader, uba::BinaryWriter & writer) Line 152 C++
[Inline Frame] UbaHost.dll!std::_Func_class<bool,uba::ConnectionInfo const &,uba::WorkContext const &,uba::MessageInfo &,uba::BinaryReader &,uba::BinaryWriter &>::operator()(const uba::ConnectionInfo &) Line 854 C++
UbaHost.dll!uba::NetworkServer::Worker::Update(uba::NetworkServer::WorkerContext & context) Line 631 C++
[Inline Frame] UbaHost.dll!uba::NetworkServer::Worker::ThreadWorker(uba::NetworkServer & server) Line 687 C++
[Inline Frame] UbaHost.dll!uba::NetworkServer::Worker::Start::__l2::<lambda_1>::operator()() Line 73 C++
[Inline Frame] UbaHost.dll!std::invoke(uba::NetworkServer::Worker::Start::__l2::<lambda_1> &) Line 1731 C++
UbaHost.dll!std::_Func_impl_no_alloc<`uba::NetworkServer::Worker::Start'::`2'::<lambda_1>,unsigned int>::_Do_call() Line 810 C++
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown
The other crashed in StorageImpl::HasCasFile()
[Inline Frame] UbaHost.dll!uba::CasKey::operator==(const uba::CasKey &) Line 31 C++
[Inline Frame] UbaHost.dll!std::equal_to<uba::CasKey>::operator()(const uba::CasKey &) Line 393 C++
[Inline Frame] UbaHost.dll!std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>::operator()(const uba::CasKey &) Line 152 C++
[Inline Frame] UbaHost.dll!std::_Hash<std::_Umap_traits<uba::CasKey,uba::StorageImpl::CasEntry,std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>,uba::Allocator<std::pair<uba::CasKey const ,uba::StorageImpl::CasEntry>>,0>>::_Find_last(const uba::CasKey &) Line 1574 C++
[Inline Frame] UbaHost.dll!std::_Hash<std::_Umap_traits<uba::CasKey,uba::StorageImpl::CasEntry,std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>,uba::Allocator<std::pair<uba::CasKey const ,uba::StorageImpl::CasEntry>>,0>>::_Find(const uba::CasKey &) Line 1213 C++
[Inline Frame] UbaHost.dll!std::_Hash<std::_Umap_traits<uba::CasKey,uba::StorageImpl::CasEntry,std::_Uhash_compare<uba::CasKey,std::hash<uba::CasKey>,std::equal_to<uba::CasKey>>,uba::Allocator<std::pair<uba::CasKey const ,uba::StorageImpl::CasEntry>>,0>>::find(const uba::CasKey &) Line 1225 C++
UbaHost.dll!uba::StorageImpl::HasCasFile(const uba::CasKey & casKey, uba::StorageImpl::CasEntry * * out) Line 1844 C++
UbaHost.dll!uba::StorageImpl::CopyOrLink(const uba::CasKey & casKey, const wchar_t * destination, unsigned int fileAttributes, bool writeCompressed, const std::function<bool __cdecl(uba::MemoryBlock &,void const *,unsigned __int64,wchar_t const *)> & formattingFunc, bool isTemp, bool allowHardLink) Line 2117 C++
UbaHost.dll!uba::SessionServer::HandleSendFileToServer(const uba::ConnectionInfo & connectionInfo, const uba::WorkContext & reader, uba::BinaryReader & writer, uba::BinaryWriter &) Line 1189 C++
UbaHost.dll!uba::SessionServer::{ctor}::__l2::<lambda_2>::operator()(const uba::ConnectionInfo & connectionInfo, const uba::WorkContext & workContext, uba::MessageInfo & messageInfo, uba::BinaryReader & reader, uba::BinaryWriter & writer) Line 152 C++
[Inline Frame] UbaHost.dll!std::_Func_class<bool,uba::ConnectionInfo const &,uba::WorkContext const &,uba::MessageInfo &,uba::BinaryReader &,uba::BinaryWriter &>::operator()(const uba::ConnectionInfo &) Line 854 C++
UbaHost.dll!uba::NetworkServer::Worker::Update(uba::NetworkServer::WorkerContext & context) Line 631 C++
[Inline Frame] UbaHost.dll!uba::NetworkServer::Worker::ThreadWorker(uba::NetworkServer & server) Line 687 C++
[Inline Frame] UbaHost.dll!uba::NetworkServer::Worker::Start::__l2::<lambda_1>::operator()() Line 73 C++
[Inline Frame] UbaHost.dll!std::invoke(uba::NetworkServer::Worker::Start::__l2::<lambda_1> &) Line 1731 C++
UbaHost.dll!std::_Func_impl_no_alloc<`uba::NetworkServer::Worker::Start'::`2'::<lambda_1>,unsigned int>::_Do_call() Line 810 C++
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown
The dump files are attached, and each contains all thread information at the time of the crash.
Thanks for the reply. We do have plans to upgrade to UE5.7 within two months, and I’ll check if it’s ok after upgrading.
A dedicate release channel for UBA sounds greate!