Recommended usage of ControlRigEntitySystem in Sequencer ( to fix visual glitches )

Hello,

We have a MasterSequence containing 2 SubSequences : Shot A and Shot B. When transitionning from Shot A to Shot B, the first frame of Shot B has glitched animation on our Characters.

On the first frame only, the pawn is invisible, and on frame 2 it comes in with heavy motion blur. It looks exactly like a teleportation.

Our Character’s animation setup is simple and pretty flat : a SkeletalAnimationTrack and a ControlRigParameterTrack.

If using KeepState on the ControlRig track, the glitch disappears. If muting the ControlRig track, we have no glitch. Also, in Editor and for the same instance of Sequencer, if we lock->unlock the SkeletalAnimationTrack, the glitch is fixed. If we close -> reopen the Sequencer, the glitch comes back.

While investigating we saw this CVAR : ControlRig.UseLegacySequencerTemplate, which is true by default. By turning it to false, the glitch disappears.

We are considering using it even though it is new. Is it safe to use ? Does this come with any drawbacks, issues or specificities ? Is there anything we should be keeping an eye on ?

Thank you

Hey there,

We are considering using it even though it is new. Is it safe to use ?

We’re planning on turning this on in 5.8. But there are issues in 5.6. It is “safe,” but you’ll most likely encounter other errors, such as being unable to animate or correctly grab some controls in the control rig.

It has been disabled internally until recently, and we are now testing all follow-up cases. Because of that, there may be further discrepancies in your 5.6 version that we aren’t seeing at the moment.

My suggestion would be to turn it on and, if you see any further issues, let us know here so we can follow up on any cherry-pick CLs. If those aren’t possible for some reason, you may want to wait for 5.8.

Dustin

Hi Dustin,

Coming back after a bit of investigation. We enabled the new ControlRigSystem and encountered some issues.

First issue was a big desync on first frame. We managed to fix it by adding in MovieSceneControlRigSystem.cpp in the UMovieSceneControlRigParameterEvaluatorSystem constructor :

DefineImplicitPrerequisite(UByteChannelEvaluatorSystem::StaticClass(), GetClass());
DefineImplicitPrerequisite(UBoolChannelEvaluatorSystem::StaticClass(), GetClass());
DefineImplicitPrerequisite(UIntegerChannelEvaluatorSystem::StaticClass(), GetClass());

This ensured that the bool/byte/int values of our CR were evaluated before the ControlRigSystem was using them.

But then we still have an issue in regards to the character’s animation on a shot transition. Always on second frame. I attached an archive containing :

- project to repro the issue using 5.6 vanilla + 3rdP Template assets

- a video showcasing the issue

We use a Sequence containing nothing but two shot sequences. In each Shot Sequence we have a Possessable with an Animation Track and a Layered FK Control Rig track.

The ABP used is simple : a “Slot” Node connected to the OutputPose. The animation is then filled by a Sequencer track.

The problem can be tracked on a bone level, for example on my actual project when tracking the head bone :

F1 : head_M_bjn V(X=-9.90, Y=1.53, Z=-15.18)

F2 : head_M_bjn V(X=-9.96, Y=1.55, Z=-14.86)

F3 : head_M_bjn V(X=-9.90, Y=1.53, Z=-15.17)

F4 : head_M_bjn V(X=-9.90, Y=1.53, Z=-15.17)

F5 : head_M_bjn V(X=-9.89, Y=1.53, Z=-15.17)

F6 : head_M_bjn V(X=-9.89, Y=1.53, Z=-15.17)

F7 : head_M_bjn V(X=-9.89, Y=1.52, Z=-15.15)

You can see the issue on frame #2.

When going more in depth :

- 1st frame seems fine from on the whole. The value for a given bone from SkelAnimation to the application of the CR is OK.

- 2nd frame starts with the wrong bone value. Even before the ControlRig is evaluated. Looking in

Plugins\Animation\ControlRig\Source\ControlRig\Private\AnimNode_ControlRigBase.cpp :
 
FAnimNode_ControlRigBase::Evaluate_AnyThread(FPoseContext& Output)
{
 
FPoseContext SourcePose(Output);
 
if (Source.GetLinkNode())
{
    Source.Evaluate(SourcePose)
}
 
...
}

the SourcePose is already wrong even though this is just the AnimationBlueprint.

- 3rd frame we’re back on track as if nothing happened and the SourcePose contains coherent data.

If this issue reminds you of anything let me know. We’ll continue to investigate on our end.

MyProject2.zip(57.2 MB)

Yeah, this looks like it’s been addressed for 5.8. There are a significant number of changes that would need to be backported to 5.6 and it might not be feasible for us to proceed. Is an engine upgrade viable in your future?

Dustin

We’re on a very tight schedule so upgrading to 5.8 won’t be possible. 5.7 might be though, would it be easier to backport on 5.7 or is the bulk of the changes from 5.7 to 5.8 ?

Also could you point to any commit where this is adressed ? Or maybe a quick explanation explaining what’s the core issue ? I might be able to figure something out

The bulk of the changes are in 5.7->5.8. It may be possible to backport, there are a significant amount of changes and fixes that have gone in since January, with most of them touching MovieSceneControlRigSystem.cpp. You can see the majority of changes here (even our backouts), there’s about 6 and you’d want to check the whole CL for what files were touched:

https://github.com/EpicGames/UnrealEngine/commits/ue5-main/Engine/Plugins/Animation/ControlRig/Source/ControlRig/Private/Sequencer/MovieSceneControlRigSystem.cpp

Basically, any CL marked “Control Rig ECS”. The basic gist is that control rig’s binding with sequencer’s ECS backing needed some maintenance as control rig changed and the evaluation phase of ECS wasn’t updated to support it.

Just thinking about your needs, is this something that you can leave on and then bake down to address the issue, but remove for shipping?

Dustin

Hey, after a quick chat with our Technical Director, even 5.7 seems compromised so upgrading doesn’t seems possible. It might not be necessary though as I found something interesting.

The visible animation hiccup is not due to anything with the ControlRig evaluation, but with the Montage played within the Sequencer. I noticed that the montages were updated twice each frame, and one of those updates produced exactly the hiccup we see.

Tracking in UAnimInstance::UpdateMontageEvaluationData(). Each frame is separated by a new line.

FAnimMontageInstance::Position: _1.707186_

FAnimMontageInstance::Position: _1.748000_

FAnimMontageInstance::Position: _1.710813_

FAnimMontageInstance::Position: _1.801498_

FAnimMontageInstance::Position: _1.712444_

FAnimMontageInstance::Position: _1.753211_

FAnimMontageInstance::Position: _1.714118_

FAnimMontageInstance::Position: _1.755984_

FAnimMontageInstance::Position: _1.715774_

FAnimMontageInstance::Position: _1.757160_

We can see in non-bold that we have a normal update, linear and with standard deltas.

And we can see a second update in bold that has the infamous 2nd frame hiccup.

So now how to find the problematic updates. The Montage updates happen in UAnimInstance::UpdateAnimation(). Below details on how it is called.

Setup #1 :

  1. An AActor with a SkeletalMeshComponent with an ABP
  2. A LevelSequence with this AActor as a Possessable
  3. Add a SkeletalAnimationTrack with an AnimationSection
  4. ( no control rig )

The SkeletalMeshComponent has one AnimInstance, which is the ABP. It is updated in :

  1. USkeletalMeshComponent::TickComponent(..)
  2. USkinnedMeshComponent::TickComponent(..)
  3. USkeletalMeshComponent::TickPose(..)
  4. USkeletalMeshComponent::TickAnimation(..)
  5. USkeletalMeshComponent::TickAnimInstances(..)
void USkeletalMeshComponent::TickAnimInstances(float DeltaTime, bool bNeedsValidRootMotion)
{
 
[...]
 
	// We update linked instances first incase we're using either root motion or non-threaded update.
	// This ensures that we go through the pre update process and initialize the proxies correctly.
	for (UAnimInstance* LinkedInstance : LinkedInstances)
	{
		// Sub anim instances are always forced to do a parallel update 
		LinkedInstance->UpdateAnimation(DeltaTime * GlobalAnimRateScale, false, UAnimInstance::EUpdateAnimationFlag::ForceParallelUpdate);
	}
 
	if (AnimScriptInstance != nullptr)
	{
		// Tick the animation
		AnimScriptInstance->UpdateAnimation(DeltaTime * GlobalAnimRateScale, bNeedsValidRootMotion);
	}
 
[...]
}

As you can see in this TickAnimInstances(..) we first update LinkedAnimInstances, then the main AnimInstance. In this first setup, we don’t have any LinkedAnimInstances so it’s fine.

Setup #2 :

  1. Use Setup#1
  2. Add a ControlRigTrack in Layered mode

Adding a ControlRigTrack creates a UControlRigLayerInstance and assigns it to the SkeletalMeshComponent. The ABP AnimInstance is added as a LinkedAnimInstance.

So now the previous callstack also happens but on the ControlRigLayerInstance which calls the ABP AnimInstance in the LinkedAnimInstances. This is fine.

But in the ControlRigLayerInstance, we then find this bit :

void UControlRigLayerInstance::NativeUpdateAnimation(float DeltaSeconds)
{
	Super::NativeUpdateAnimation(DeltaSeconds);
	
	if (UAnimInstance* SourceAnimInstance = GetSourceAnimInstance())
	{
		if (UControlRig* ControlRig = GetFirstAvailableControlRig())
		{
			if (ControlRig->IsAdditive())
			{
				SourceAnimInstance->UpdateAnimation(DeltaSeconds, true, EUpdateAnimationFlag::Default);
			}
		}
	}
}

SourceAnimInstance is the ABP AnimInstance. So the ABP AnimInstance is updated through the usual way, but also another time with this call, leading to 2 AnimMontage updates, creating our problem.

Commenting this fixes our issue. This code is there since 5.4. Looking at Unreal’s GitHub, it is still there. Is it still relevant ? Can we get rid of it safely ? Do you have any tips/knowledge to share about this ?

Relevant commit adding it : https://github.com/EpicGames/UnrealEngine/commit/c01e888073a912988bbda178914f01cd4849faae

Thank you !

Hey,

Passing it to 0.0f does indeed also fix the issue. Thank you will use this one

But just fyi, our Layered CRs still worked when commenting the SourceAnimInstance->UpdateAnimation(..) in UControlRigLayerInstance::NativeUpdateAnimation. The whole reason this double update even happens is that SourceAnimInstance can be found in the LinkedAnimInstances of the owning SkelMeshComponent, so is updated in USkeletalMeshComponent::TickAnimInstances(

) before the AnimScriptInstance ( Layered CR ) is.

I see that the two calls have different flags so I guess there’s a need for it ? We just have not found it and didn’t have time to investigate that much further

USkeletalMeshComponent::TickAnimInstances(..)

- LinkedInstance->UpdateAnimation(DeltaTime * GlobalAnimRateScale, false, UAnimInstance::EUpdateAnimationFlag::ForceParallelUpdate);

UControlRigLayerInstance::NativeUpdateAnimation()

- SourceAnimInstance->UpdateAnimation(DeltaSeconds, true, EUpdateAnimationFlag::Default);

Looked a bit further and :

- A ControlRigLayerInstance by default has bUseMultiThreadedAnimationUpdate = false ( see UControlRigLayerInstance::UControlRigLayerInstance - ControlRigLayerInstance.cpp l.38 )

- When adding a ControlRig track, if additive, its SourceAnimInstance is also bUseMultiThreadedAnimationUpdate = false ( see UControlRigLayerInstance::AddControlRigTrack - ControlRigLayerInstance.cpp l.118 )

- When UAnimInstance::UpdateAnimation is called, we update immediately only if EUpdateAnimationFlag::Default is passed in and with UAnimInstance::NeedsImmediateUpdate returning true ( l.652 ).

- When looking into UAnimInstance::NeedsImmediateUpdate, we can see that if bUseMultiThreadedAnimationUpdate is false and CVarForceUseParallelAnimUpdate is false ( it is by default ), then it returns true.

So when in USkeletalMeshComponent::TickAnimInstances(..), all LinkedInstances have the flag ForceParallelUpdate. This updates montage playback values only.

Afret that ( CRLayerInstance ) AnimScriptInstance->UpdateAnimation is called with the Default flag + NeedsImmediateUpdate returning true. Updating montage playback values. From the UControlRigLayerInstance::NativeUpdateAnimation call, it also updates its SourceAnimInstance with Default flag + NeedsImmediateUpdate also returning true. Updating montage playback values.

Then SourceAnimInstance calls ParallelUpdateAnimation() and then AnimScriptInstance calls it.

We end up with two montages updates on the SourceAnimInstance. One on the CRLayerInstance. And one ParallelUpdateAnimation() on both.

That’s great and all but in our fix, we commented the immediate update coming from the CRLayerInstance, and it fixed the issue. That means we didn’t have the second montage update ( the real fix ), but as a side effect, we also lost the ParallelUpdateAnimation() on the SourceAnimInstance. Yet, it seems to work flawlessly. Why ?

This suggests that the UControlRigLayerInstance::NativeUpdateAnimation is simply not needed.

This got me curious as to why we need immediate update at all and we still had the AnimScriptInstance->UpdateAnimation() call that did immediate update. So I enabled CVarForceUseParallelAnimUpdate. And it still works great.

The code seems to suggest that when using additive controlrig, we cannot use multithreaded work, and we need immediate work. Our experiment means we don’t really ? Am I missing something ? Is it because our use case is Sequencer only ? Does it have to do with RootMotion ?

Awesome, making a lot more sense. Thanks for the detailed explanation !