After upgrading to 5.6 we’re seeing frequent deadlocks in what appears to be animation compression. It’s not reproable, presumably because some results are cached and thus won’t need building causing the (again, presumed) race condition.
[Image Removed]
Steps to Reproduce
Looking closer it seems it might be a problem with uploading to the derived data cache
Hi Michael,
Thanks for your question, are you saying that there is something on your side that’s causing this? If not, would you be able to provide the wider parallel stacks, and also the assets involved (and their relationship). E.g. is the worker thread building animation data for a Montage referenced animation? (as far as I can deduce from the provided image)
Cheers,
Jurre
Various animation assets trigger the deadlock, but only once; after making it to DDC it will not happen again for that asset. These are screenshots from a colleague; I’ll try get better info next time it happens.
[Image Removed]
This is the deadlock as far as I can tell: Thread 3044 is waiting for the SharedCompressedDataMutex of the anim sequence A_Civilian_M_Run_Generic2, which is helt by thread 10364. I can’t tell exactly what 10364 is waiting for, but it would seem to be a barrier or lock helt by 3044.[Image Removed]
This deadlock also has a 100% repro when selecting “Apply compression” for a particular AnimSequence.[Image Removed]
I had a look at this and something worth trying is moving UE_COMPRESSED_DATA_WRITE_SCOPE inside of UAnimSequence::FinishAsyncTasks to be opened after It->Value->Wait**()**; as such:
`void UAnimSequence::FinishAsyncTasks()
{
const ITargetPlatform* RunningPlatform = GetTargetPlatformManagerRef().GetRunningTargetPlatform();
const FIoHash CurrentPlatformHash = CreateDerivedDataKeyHash(RunningPlatform);
const bool bHasInflightTasks = CacheTasksByKeyHash.Num() > 0;
if (bHasInflightTasks)
{
COOK_STAT(auto Timer = UE::Anim::AnimSequenceCookStats::UsageStats.TimeAsyncWait());
COOK_STAT(Timer.TrackCyclesOnly());
for (auto It = CacheTasksByKeyHash.CreateIterator(); It; ++It)
{
It->Value->Wait();
UE_COMPRESSED_DATA_WRITE_SCOPE(this); <<<
FCompressedAnimSequence* TaskData = It->Value->GetTargetCompressedData();
const bool bCompressedRunningPlatform = It->Key == CurrentPlatformHash;
auto ResetData = this, TaskData, It
{
if(TaskData != nullptr)
{
TaskData->Reset();
}
DataByPlatformKeyHash.Remove(It->Key);
};`
Michael Lidgren - were you able to take a look at the above proposed fix?
(sorry for slow reply; vacations)
Yes, this change does indeed solve my 100% repro! Thank you!
No worries, I had a feeling my answer was overlapping with Swedish Summer break dates! In that case the fix will be integrated into mainline- thanks for reporting and the help!