Pose Search DDC Questions

Development Character & Animation
, ,
1

Some questions about Pose Search indexing. We’ve experienced a couple issues recently and I wanted to get a bit more light on them.

The first thing is that when we launch the editor we always get logs like these:

```

[0219.69][309]LogPoseSearch: 0fdb9631774230e2f1049ea417a53b88ca3968ea - PSD_Character_Item_Loco_Idles BeginCache

[0219.69][309]LogPoseSearch: 0fdb9631774230e2f1049ea417a53b88ca3968ea - PSD_Character_Item_Loco_Idles BuildIndex From Cache

[0219.76][313]LogPoseSearch: dc989331923e55abb5e38e86f954f65daa99b8c2 - PSD_Character_Item_Loco_Loops BeginCache

[0219.76][313]LogPoseSearch: dc989331923e55abb5e38e86f954f65daa99b8c2 - PSD_Character_Item_Loco_Loops BuildIndex From Cache

[0219.85][319]LogPoseSearch: Delaying DDC until dependents post load - PSD_Character_Item_Loco_Pivots

[0219.88][319]LogPoseSearch: Delaying DDC until dependents post load - PSD_Character_Item_Loco_Pivots

[0219.95][320]LogPoseSearch: c77b31c6ae806353b8811c6cd81dfd49bc8f83f4 - PSD_Character_Item_Loco_Pivots BeginCache

[0219.95][320]LogPoseSearch: c77b31c6ae806353b8811c6cd81dfd49bc8f83f4 - PSD_Character_Item_Loco_Pivots BuildIndex From Cache

[0221.01][384]LogPoseSearch: 9e1c7f2c1a411483e29050911f1f622958d3cfcf - PSD_Character_Regular_Jog_Loops BeginCache

[0221.01][384]LogPoseSearch: 9e1c7f2c1a411483e29050911f1f622958d3cfcf - PSD_Character_Regular_Jog_Loops BuildIndex From Cache

[0221.11][391]LogPoseSearch: Delaying DDC until dependents post load - PSD_Character_Regular_Jog_Pivots

[0221.12][391]LogPoseSearch: Delaying DDC until dependents post load - PSD_Character_Regular_Jog_Pivots

[0221.17][392]LogPoseSearch: c74e88a385ac7dcbc35183f4fb9b6f08dc3ecf93 - PSD_Character_Regular_Jog_Pivots BeginCache

[0221.17][392]LogPoseSearch: c74e88a385ac7dcbc35183f4fb9b6f08dc3ecf93 - PSD_Character_Regular_Jog_Pivots BuildIndex From Cache

[0221.20][394]LogPoseSearch: bb6797330a90c4b6dc7207c8801ea1d57b5752ed - PSD_Character_Regular_Sprint_Loops BeginCache

[0221.20][394]LogPoseSearch: bb6797330a90c4b6dc7207c8801ea1d57b5752ed - PSD_Character_Regular_Sprint_Loops BuildIndex From Cache

[0221.22][396]LogPoseSearch: Delaying DDC until dependents post load - PSD_Character_Regular_Sprint_Pivots

[…]

[0231.62][172]LogPoseSearch: Delaying DDC until dependents post load - PSD_XXX (we get this for various databases)

```

Is it expected for indices to be built when we launch the editor?

A second issue we had was that we changed the skeleton on some animations, but forgot to change it on the Pose search schema, this lead to the cook sometimes stalling with a message like this:

```

[2025-07-14T05:55:55.319Z] LogCook: Display: Cooked packages 9868 Packages Remain 2 Total 9870

[2025-07-14T05:55:55.319Z] LogCook: Warning: Cooker has been blocked from saving the current packages for 16321 seconds.

[2025-07-14T05:55:55.319Z] LogCook: Display: 2 packages in the savequeue:

[2025-07-14T05:55:55.319Z] LogCook: Display: /Game/GameSystems/…/PSD_1

[2025-07-14T05:55:55.319Z] LogCook: Display: /Game/GameSystems/…/PSD_2

[2025-07-14T05:55:55.319Z] LogCook: Display: 2 objects that have not yet returned true from IsCachedCookedPlatformDataLoaded:

[2025-07-14T05:55:55.319Z] LogCook: Display: PoseSearchDatabase /Game/GameSystems/…/PSD_1.PSD_1

[2025-07-14T05:55:55.576Z] LogCook: Display: PoseSearchDatabase /Game/GameSystems/…/PSD_2.PSD_2

```

I believe correcting the schemas fixed this issue, but it wasn’t happening 100% either. Thought I’d mention it in case it’s something you’d want to track.

2

Hi, yeah, the log messages about the key being generated on editor startup, etc, are expected. The idea is to make sure that there’s a valid DDC index before you get to the point of running PIE, otherwise we could potentially have to wait at that point, which could mean a number of frames without a valid pose.

In terms of the second issue with the cooker being blocked, do you have any more specific repro steps or was it just changing the skeleton on the anim sequences and re-cooking? Do you remember how consistently you’d run into the problem? We’ve had some reports previously of similar errors and we’ve made some fixes, but not to anything related to dependent skeletons, so I’d like to try and get a repro on that if possible.

3

Hi Euan,

Regarding cooks being softlocked, we managed to get ourselves into a 100% repro frequency situation. We have unblocked us with a workaround but would like to discuss our findings to gather your thoughts on them.

Our cooks ended up softlocked when a PoseSearchDatabase asset had an associated FPoseSearchDatabaseAsyncCacheTask that had been put in the pre-cancelled state. In particular, the task had been pre-cancelled when some other PoseSearchDatabase asset was saved as part of the normal cooking process (as a byproduct of FPoseSearchDatabaseAsyncCacheTask::CancelIfDependsOn). We noticed that tasks put in the pre-cancelled state expect to be cancelled at some point in the future (specifically when FAsyncPoseSearchDatabasesManagement::ClearPreCancelled is called), but cancellation only happens if the FCoreUObjectDelegates::OnObjectTransacted or FCoreUObjectDelegates::OnObjectPropertyChanged delegates are triggered (as per FAsyncPoseSearchDatabasesManagement’s code in 5.5), and the cook process was able to reach a point where none of those delegates would be invoked, thus leading to the softlock.

Is there a reason why FAsyncPoseSearchDatabasesManagement::ClearPreCancelled() is not periodically called to clean-up pre-cancelled tasks? (e.g. in FAsyncPoseSearchDatabasesManagement::Tick) As far as we can tell from the code there does not appear to be an implicit dependency that would mean that ClearPreCancelled() is only safe to be called from those delegates (provided its execution happens on the game thread and the internal mutex is locked normally)

4

Tangent to Pablo’s investigation. It seems like Sync with external dependencies almost always calls Modify() during cook. Which I’d expect would only happen if the list of entries has changed. Would this mean that we have failed to properly resave that pose search database after the animations have changed?

Also if we update the branch in sampling range of an existing entry, that doesn’t mark bModified as true, and it also doesn’t change the BranchInId since that only uses the notify GetFullName, which does not encode the sampling range. Shouldn’t that call Modify too? In case there’s only been an update on the sampling range? Wouldn’t it be best to include the sampling range in the BranchInId hash?

5

Hi [mention removed]​, yeah so the idea with this code is that the delegates bound to OnObjectTransacted and OnObjectPropertyChanged should be hit for whatever dependent asset has changed. And those functions should result in the task being cancelled.

So the question is why neither of those is being hit when you run into the soft lock (but FAsyncPoseSearchDatabasesManagement::OnPreObjectPropertyChanged or FAsyncPoseSearchDatabasesManagement::OnObjectModified are being hit to request the cancellation). Are you able to get a callstack for the call to FPoseSearchDatabaseAsyncCacheTask::PreCancelIfDependsOn that results in the soft lock? That might help to see if there is a codepath that’s missing the symmetrical calls to OnObjectTransacted or OnObjectPropertyChanged.

6

Hi, this is the relevant part of the callstack:

>	UnrealEditor-PoseSearch.dll!UE::PoseSearch::FPoseSearchDatabaseAsyncCacheTask::PreCancelIfDependsOn(const UObject * Object) Line 1515	C++
	UnrealEditor-PoseSearch.dll!UE::PoseSearch::FAsyncPoseSearchDatabasesManagement::PreModified(UObject * Object) Line 2164	C++
	[Inline Frame] UnrealEditor-PoseSearch.dll!Invoke(void(UE::PoseSearch::FAsyncPoseSearchDatabasesManagement::*)(UObject *)) Line 66	C++
	[Inline Frame] UnrealEditor-PoseSearch.dll!UE::Core::Private::Tuple::TTupleBase<TIntegerSequence<unsigned int>>::ApplyAfter(void(UE::PoseSearch::FAsyncPoseSearchDatabasesManagement::*)(UObject *) &) Line 317	C++
	UnrealEditor-PoseSearch.dll!TBaseRawMethodDelegateInstance<0,UE::PoseSearch::FAsyncPoseSearchDatabasesManagement,void __cdecl(UObject *),FDefaultDelegateUserPolicy>::ExecuteIfSafe(UObject * <Params_0>) Line 536	C++
	[Inline Frame] UnrealEditor-CoreUObject.dll!TMulticastDelegateBase<FDefaultDelegateUserPolicy>::Broadcast(UObject *) Line 257	C++
	UnrealEditor-CoreUObject.dll!TMulticastDelegate<void __cdecl(UObject *),FDefaultDelegateUserPolicy>::Broadcast(UObject * <Params_0>) Line 1079	C++
	[Inline Frame] UnrealEditor-CoreUObject.dll!FCoreUObjectDelegates::BroadcastOnObjectModified(UObject * Object) Line 3115	C++
	UnrealEditor-CoreUObject.dll!UObject::Modify(bool bAlwaysMarkDirty) Line 1520	C++
	UnrealEditor-PoseSearch.dll!UPoseSearchDatabase::SynchronizeWithExternalDependencies(TArrayView<UAnimSequenceBase * const,int> SequencesBase) Line 982	C++
	UnrealEditor-PoseSearch.dll!UPoseSearchDatabase::SynchronizeWithExternalDependencies() Line 849	C++
	UnrealEditor-PoseSearch.dll!UPoseSearchDatabase::PreSaveRoot(FObjectPreSaveRootContext ObjectSaveContext) Line 1027	C++
	UnrealEditor-CoreUObject.dll!UE::SavePackageUtilities::CallPreSaveRoot(UObject * Object, FObjectSaveContextData & ObjectSaveContext) Line 699	C++
	UnrealEditor-CoreUObject.dll!UPackage::Save2(UPackage * InPackage, UObject * InAsset, const wchar_t * InFilename, const FSavePackageArgs & SaveArgs) Line 3199	C++
	UnrealEditor-CoreUObject.dll!UPackage::Save(UPackage * InOuter, UObject * InAsset, const wchar_t * Filename, const FSavePackageArgs & SaveArgs) Line 20	C++
	UnrealEditor-UnrealEd.dll!UEditorEngine::Save(UPackage * InOuter, UObject * InAsset, const wchar_t * Filename, const FSavePackageArgs & InSaveArgs) Line 4434	C++
7

Hi [mention removed]​, I’ll need to confirm with the dev team tomorrow but both of these issues look like bugs related to the change on the previous EPS thread. Because we’re always removing and re-adding the existing assets, the DB is always going to be marked as having been modified. That doesn’t seem necessary to me.

The point around bModified not being set when the sampling range changes also looks to be correct. We should be re-processing the database after that happens and it looks like we aren’t. I’ll follow up once I have more info.

8

Hi Euan,

Just following up to see if there are any updates on the cook stalls issue.

Cheers

9

Hi Sergio, no update just yet. It was a long weekend here last weekend so that’s slowed things down. I’m aiming to sync up with the dev team further about this tomorrow and I’ll get back to you with next steps after that.

10

Quick update - sorry again for the delay. I spent some time going through the branch in synchronization code with the dev team last week, and we have a potential fix for both of the issues you’ve been running into, but there are some issues applying it to 5.4. I’ll need to dig into those further tomorrow. I’ll get back to you once I have more info.

11

Thanks Euan, we’re on 5.5 in case that simplifies the integration.

12

Hi [mention removed]​ [mention removed]​, we have a potential fix for both issues that I’ve backported to 5.5 for you to test. It’s shelved at 44769485. If you’d like a patch instead of a shelf, just let me know. The equivalent fix, along with a fix for another issue with pose search DB synchronization when assets are transient, went into UE5/Main at 44764511. The fix for the transient assets is more involved, so unless you run into that issue (ie. adding and later removing assets to a database before the asset has been saved), then I think you should be good with just the changes in my shelf.

The changes modify UPoseSearchDatabase::SynchronizeWithExternalDependencies so that we now have a temporary array of assets that we add and remove assets from. We then compare that with the original array and apply it if anything has changed. The changes also add a transaction scope into UPoseSearchDatabase::PreSaveRoot. That not being present looks to have been the cause of the issue with the task not having been cancelled correctly after being set to pre-cancelled. Some code paths that called SynchronizeWithExternalDependencies were adding a transaction scope, but the save code path wasn’t.

I’ve spent a bit of time testing these changes with simple setups, but I’d strongly recommend testing them thoroughly with your assets to make sure there are no issues.

I also wanted to give you a heads up that longer term at least some of the branch-in functionality is likely to change. Once we have a motion matching implementation working with choosers (ie. a pose search column that can perform selection within a chooser), the need for notifies to perform the synchronization of assets with a database is likely no longer required since the chooser can serve a similar purpose. At that point the branch in notifies will just be used to mark the frame range to sample from. That’ll allow us to remove the synchronization code which is fragile as you’ve seen.

13

Hey there -- also seeing this in 5.6. [mention removed]​ You said you had a workaround?

It’s intermittently occurring on our Horde builders when cooking -- soft locks in the cooker waiting for it to save, until someone or Horde eventually cancels the build. Always related to PSD files, but not always the same ones. And sometimes it works fine.

Looking at pulling in 44438059, and 44764511 or 44769485 .. any others that might be related?

14

We had one but we’ve adopted the changes in CL 44769485 and removed the workaround since then - no issues so far.

If you’re concerned about the surface area of the changes in that CL, and your softlocks happen to have the exact same cause our ones had (tasks pre-cancelled in the context of UPoseSearchDatabase::PreSaveRoot never progressing to the cancelled state) you could try just adding a transaction scope to this function - this is part of the changes to PoseSearchDatabase.cpp in CL 44769485. Specifically:

// ... top of the file
 
#if WITH_EDITOR
// BEGIN CHANGE
#include "ScopedTransaction.h"
// END CHANGE
#include "AssetRegistry/AssetRegistryModule.h"
#endif //WITH_EDITOR
 
// ...
 
DEFINE_STAT(STAT_PoseSearch_Event);
 
// BEGIN CHANGE
#define LOCTEXT_NAMESPACE "PoseSearchDatabase"
// END CHANGE
 
namespace UE::PoseSearch
 
// ...
 
void UPoseSearchDatabase::PreSaveRoot(FObjectPreSaveRootContext ObjectSaveContext)
{
#if WITH_EDITOR
// BEGIN CHANGE
	FScopedTransaction ScopedTransaction(LOCTEXT("PoseSearchDatabaseSynchronizeWithExternalDependencies", "Pose Search Database Synchronize With External Dependencies"));
// END CHANGE
	// in case the database desynchronized with the UAnimNotifyState_PoseSearchBranchIn referencing it, we need to resynchronize
	SynchronizeWithExternalDependencies();
#endif
 
	Super::PreSaveRoot(ObjectSaveContext);
}

We haven’t tried this in isolation so no guarantees, but AFAICT it should have the same effect our previous workaround had (for that specific problem).

15

Yeah [mention removed]​, if the issue is specifically that the tasks aren’t being cancelled correctly, then just the change to add the scoped transaction to PreSaveRoot should fix it.

16

Neither of the provided CLs apply cleanly on their own, so attempting just the minimal version as described, will report back after we’ve had some time to put it through it’s paces. Thanks for cutting it down to the minimum. It’s difficult for me to say if that was specifically the problem, since it’s rather difficult to debug intermittent problems on Horde builders, but it seems pretty likely, and applying that will tell us pretty quickly if so.

edit: noting that there needs to be a #undef LOCTEXT_NAMESPACE at the end of the file

a first test run cooking for a platform i’ve never cooked for, so i had zero DDC for, seems to have been successful, so going to put this into our system and run with it

17

Unfortunately, that did not solve the issue, and I think we’re going to probably discontinue use of the plugin until we can get it stable.

Or perhaps do we need to resave the PSDs after applying this for it to work? Might maybe try it, and possibly see if I can nuke whatever data is on the Zen server for them, in case maybe that’s messed up too.

Kinda spitballing ideas here, since I don’t really know a whole lot about the internals of cooker process, I’m just trying to get my build systems to be stable :slight_smile:

18

Yes, I’d recommend resaving all PSDs to see if the issue is resolved. It’s possible you’re changing notify state windows which generates the need for synchronizing with external dependencies during cook, as the PSD entries are no longer up to date. Resaving the PSD should solve this.

19

[mention removed]​ since you’re still running into problems with this, I’d recommend opening a new EPS thread where we can discuss it further there. It may be that you’re hitting a different problem to the one original discussed on this thread. If you have any logs from when the lock occurs, those could be a good place to start.

20

Unfortunately, the only thing I’m seeing is

> LogCook: Warning: Cooker has been blocked from saving the current packages for XXXXX seconds

along with the messages about which files. Nothing any more specific about it. We’ll try a resave today. Thanks