Layered Blend Per Bone nodes evaluate to 0.0 weight with Blend Masks when s.AsyncLoadingThreadEnabled is active in Editor

Development Platform & Builds
, ,
1

Hi!

When starting the Editor, Layered Blend Per Bone nodes using Blend Masks inside dynamically linked Anim Layers completely ignore the blend pose (evaluating to 0.0 weight for all bones). Disconnecting/reconnecting and recompiling fixes it for the session, but it breaks again on the next Editor boot. This does not happen if s.AsyncLoadingThreadEnabled is disabled.

From what I’ve been able to find out, it seems there’s an issue related to the BlendProfile’s PostLoad. It might not be exactly that, but I’ll share my reasoning and how I managed to track down the bug, in case it’s of any help to you

By debugging the engine source code, we caught the exact moment of failure. During the asynchronous loading phase inside FAnimationRuntime::CreateMaskWeights(), the pointer to the UBlendProfile asset is not null, but we inspected its object flags:

BlendMask->HasAnyFlags(RF_NeedPostLoad) returns TRUE

BlendMask->HasAnyFlags(RF_LoadCompleted) returns TRUE

We have some ABPs that work and others that don’t; in the ones that work, RF_NeedPostLoad returns false, and in the ones that don’t, it returns true.

Because the UBlendProfile is still loading (I think) and hasn’t executed its PostLoad() yet, its internal FBoneReference entries have their BoneName populated correctly, but their BoneIndex is still uninitialized at -1.

The FAnimationRuntime::CreateMaskWeights() then caches the compiled weights as 0.0 for all bones due to those -1 indices, marks the node’s skeleton GUID as valid, and the node locks itself in a broken state for the rest of the session.

I’ve tried setting it up so that if the blendmask hasn’t run the postload, it sets the SkeletonGuid and VirtualBoneGuid to invalid values, so that they are created during the next rebuild, and it seems to work well.

It might not be exactly this, but it’s what we’ve reproduced; I understand that, as async loading is experimental, there may be minor bugs of this sort.

Thanks,

Javi

[Attachment Removed]

2

Hey there,

I’m going to raise this with the dev team to see if we can get you a more definitive answer, but another option you might try is to run ConditionalPostLoad() on the BlendProfiles on the skeleton postload. This might guarantee the dependency order loading here.

In USkeleton::PostLoad()

for (UBlendProfile* BlendProfile : BlendProfiles)
{
	BlendProfile->ConditionalPostLoad();
}

Dustin

[Attachment Removed]

3

Hey there,

I believe this might actually be related to a different issue where bone caching fails for certain nodes in the anim graph in a linked layer scenario. If you don’t have this as a linked layer, does the issue persist? Would you be able to share a sample of your setup? This would help us pinpoint the issue.

Dustin

[Attachment Removed]

4

Hi Dustin,

Yes, it happens specifically with linked layers. (We have other characters using layered blend per bone nodes in a anim graph without linked layers and there are no problems)

Unfortunately, I cannot easily share our current setup, It’s going to take me some time that I don’t have. I’ll try to tell you what I saw, to see if that helps you a bit. I believe the compilation manager evaluates the graph nodes on a worker thread before the dependent UBlendProfiles have a chance to run their main-thread PostLoad() and initialize their bone indices.

During the Editor load sequence, the FAsyncLoadingThread runs FBlueprintCompilationManagerImpl::FlushCompilationQueueImpl(), which triggers ValidateAnimNodeDuringCompilation() for the linked layer.

Inside FAnimationRuntime::CreateMaskWeights(), we placed breakpoints and inspected the state of the UBlendProfile at that exact microsecond on the async thread:

BlendMask->HasAnyFlags(RF_NeedPostLoad) is TRUE.

Because its PostLoad() hasn’t executed yet on the Game Thread, all internal FBoneReference entries have their BoneName populated correctly, but their BoneIndex is still uninitialized at -1 (INDEX_NONE).

As a result, CreateMaskWeights() evaluates the weights for all bones to 0.0. Right after that, the node caches this broken state by storing the SkeletonGuid, locking itself up for the rest of the editor session. When PIE starts, the node thinks its cache is valid and never rebuilds the weights again.

For our specific scenario, when a character spawns, a property (its definition) is expected to be replicated, and in the definition’s OnRep event we perform an asynchronous load using the AssetManager for its Skeleton, ABP and AnimationBlueprintLayer. Once everything has loaded, we set the mesh, the animInstance and the AnimClassLayers.

The thing is, we have many different characters, so we have a common ABP and almost all the specific logic is in these Layers.

Hope this helps to track down the issue!

If there are any further tests, queries or debugging issues, I can have a look – it won’t take me long.

Javi

[Attachment Removed]

5

Hello, we’re also using async loading in the editor and have been encountering a similar issue. One workaround is to use BlendProfiles but that’s not always viable or convenient. After debugging I found out that in editor, UAnimGraphNode_LayeredBoneBlend::ValidateAnimNodeDuringCompilation can be called way earlier than FReferenceSkeleton inside USkeleton initializes it’s bone cache (BoneContainers), yet the RebuildPerBoneBlendWeights (called from the Validate function) sets it’s SkeletonGuid to a valid one, making ArePerBoneBlendWeightsValid erroneously return true. I don’t know the exact reason why some skeletons initialize before and some after, but the fix was to write the RebuildPerBoneBlendWeights like this:

void FAnimNode_LayeredBoneBlend::RebuildPerBoneBlendWeights(const USkeleton* InSkeleton)
{
	if (InSkeleton)
	{
		if (BlendMode == ELayeredBoneBlendMode::BranchFilter)
		{
			FAnimationRuntime::CreateMaskWeights(PerBoneBlendWeights, LayerSetup, InSkeleton);
		}
		else
		{
			/** When running in PIE, it can happen that the ref skeleton had not been loaded at the moment where our
			 * BlendMasks's bone indices were getting filled in, resulting in invalid indices and subsequently empty PerBoneBlendWeights.
			 * This function would've wrongly initialized SkeletonGuid and VirtualBoneGuid, later making the re-caching
			 * mechanism in UpdateCachedBoneData not detect the dirty state and continue with empty PerBoneBlendWeights.
			 */
			if (AreCachedBlendMasksValid() == false)
				return;
			FAnimationRuntime::CreateMaskWeights(PerBoneBlendWeights, BlendMasks, InSkeleton);
		}
 
		SkeletonGuid = InSkeleton->GetGuid();
		VirtualBoneGuid = InSkeleton->GetVirtualBoneGuid();
	}
}

with newly added

bool FAnimNode_LayeredBoneBlend::AreCachedBlendMasksValid() const
{
	if (BlendMode != ELayeredBoneBlendMode::BlendMask)
	{
                // When using BlendProfiles, blend masks should not be initialized or valid.
		// We always consider them invalid in case the mode was switched,
		// and we still have them initialized in memory.
		return false;
	}
	
	for (int32 MaskIndex = 0; MaskIndex < BlendMasks.Num(); ++MaskIndex)
	{
		const UBlendProfile* BlendMask = BlendMasks[MaskIndex];
		if (!BlendMask || BlendMask->Mode != EBlendProfileMode::BlendMask)
		{
			continue;
		}
		for (int32 EntryIndex = 0; EntryIndex < BlendMask->GetNumBlendEntries(); EntryIndex++)
		{
			if (BlendMask->ProfileEntries[EntryIndex].BoneReference.BoneIndex == INDEX_NONE)
				return false;
		}
	}
	return true;
}

This ensures that the node considers it’s blend weights invalid when they’re actually invalid and rebuilds them later on in UpdateCachedBoneData (as it would in a cooked build).

[Attachment Removed]

6

Apologies for the long delay on this,

I’ve logged a public issue, as we still haven’t found a solution to the issue, but if your workaround works for you, then that’s good. I’ve added that as info for us to track internally.

https://issues.unrealengine.com/issue/UE-391440 - it may take a 24hours for this to show.

Dustin

[Attachment Removed]

7

Hey Dustin,

That doesn’t seem to be working. In case it helps, what I do is load the visual asset bundle for a character asynchronously using the AssetManager, and that asset bundle contains the Skeleton and the ABP/Layer. Once everything has loaded, that’s when I set the skeleton, the ABP and call LinkAnimClassLayers.

Javi

[Attachment Removed]

8

Hi Dustin,

We’ve been using the workaround I mentioned all this time, and we haven’t had any related issues since. So for now, we’re sticking with that.

Thanks for following up.

Javi.

[Attachment Removed]