Misalignment causing a rare crash (check fail) in dev build when copying transform animation attribute in control rig

Development Character & Animation
, ,
1

We also got this error which helps understand what is happening.

appError called: [ASSERT] Assertion failed: IsAligned(PropertyDest, TheCppStructOps->GetAlignment()) [File:D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp] [Line: 3886]

Destination address for property does not match requirement of 16 byte alignment for /Script/Engine.TransformAnimationAttribute

So as far as I could understand this.

We end up copying attributes because the root motion delta is on the root and that bone is also in the ctrl rig, it also is the right type (TransformAnimationAttribute).

In

void FAnimNode_ControlRigBase::ExecuteControlRig

we create an mesh attribute container which uses the FDefaultAllocator which does not seem to have anything as far as alignment guaranties.

But the buffer we copy from is a FStackAttributeContainer which is 16 bit aligned.

A possible fix would be in UnrealEngine\Engine\Source\Runtime\Engine\Public\Animation\AttributesRuntime.h

to use TAlignedHeapAllocator<16> instead of FDefaultAllocator for the FMeshAttributeContainer

2

Steps to Reproduce
This is a very rare crash and we dont have clear repro steps.

It happened in dev PC build.

Best I can tell is that we have a control rig on top of the anim bp and lower we were doing a transition back to locomotion (motion matching)

3

Callstack

[FATAL] Unhandled Exception: 0x00004000

[Callstack] 0x00007ffa01ba73fa KERNELBASE.dll!UnknownFunction []

[Callstack] 0x00007ff6295c3f2e Realms.exe!FWindowsErrorOutputDevice::Serialize() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Core\Private\Windows\WindowsErrorOutputDevice.cpp:86]

[Callstack] 0x00007ff632d8285f Realms.exe!FSentryErrorOutputDevice::Serialize() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Plugins\Sentry\Source\Sentry\Private\SentryErrorOutputDevice.cpp:23]

[Callstack] 0x00007ff6293e7802 Realms.exe!FOutputDevice::LogfImpl() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Core\Private\Misc\OutputDevice.cpp:81]

[Callstack] 0x00007ff62931c0cf Realms.exe!AssertFailedImplV() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Core\Private\Misc\AssertionMacros.cpp:193]

[Callstack] 0x00007ff62931d4bd Realms.exe!FDebug::CheckVerifyFailedImpl2V() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Core\Private\Misc\AssertionMacros.cpp:984]

[Callstack] 0x00007ff62931d3f8 Realms.exe!FDebug::CheckVerifyFailedImpl2() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Core\Private\Misc\AssertionMacros.cpp:1007]

[Callstack] 0x00007ff629a99e3a Realms.exe!UScriptStruct::InitializeStruct() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp:3883]

[Callstack] 0x00007ff62df499f0 Realms.exe!UE::Anim::TWrappedAttribute<TSizedDefaultAllocator<32> >::Allocate() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Engine\Classes\Animation\WrappedAttribute.h:79]

[Callstack] 0x00007ff632639530 Realms.exe!UE::Anim::TAttributeContainer<FMeshPoseBoneIndex,TSizedDefaultAllocator<32> >::CopyFrom<FCompactPoseBoneIndex,TMemStackAllocator<0> >() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Engine\Classes\Animation\AttributesContainer.h:180]

[Callstack] 0x00007ff6363e3f92 Realms.exe!FAnimNode_ControlRigBase::ExecuteControlRig() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Plugins\Animation\ControlRig\Source\ControlRig\Private\AnimNode_ControlRigBase.cpp:247]

[Callstack] 0x00007ff6363dfc05 Realms.exe!FAnimNode_ControlRigBase::Evaluate_AnyThread() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Plugins\Animation\ControlRig\Source\ControlRig\Private\AnimNode_ControlRigBase.cpp:229]

[Callstack] 0x00007ff6363dfa27 Realms.exe!FAnimNode_ControlRig::Evaluate_AnyThread() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Plugins\Animation\ControlRig\Source\ControlRig\Private\AnimNode_ControlRig.cpp:211]

[Callstack] 0x00007ff62efc93d7 Realms.exe!FPoseLink::Evaluate() [D:\ba\w\2ae794e2c3278074\UnrealEngine\Engine\Source\Runtime\Engine\Private\Animation\AnimNodeBase.cpp:393]

4

Hi, as far as I can tell from looking through the code, the alignment behaviour with FDefaultAllocator depends on the underlying platform allocator. Do you know which allocator you’re using in your dev PC builds? If not, it should be listed in the log. Search for “Allocator:” and you should find it. Once I have that, I can take a look at exactly what the allocator is doing with the alignment.

It’s easier with FStackAttributeContainer since that should always be using TMemStackAllocator, and there, in FMemStackBase::PushBytes we either use 8 or 16 byte depending on the size of the allocation. For a transform that should always result in 16 byte alignment, as you noted.

Also, did you already try switching the allocator used by FMeshAttributeContainer to TAlignedHeapAllocator<16> or was that more speculation about a possible fix?

5

As far as I’m aware, the bins that MallocBinned2 uses are 16-byte-aligned by default, which makes it odd that you’re running into that check failure. In theory, the implementation of TWrappedAttribute::StructMemory does seem potentially problematic, as it uses unit8 as a type, even though we’ll be using that memory to store different script struct types. So the alignment that is requested in TWrappedAttribute::Allocate will often be incorrect. But in practice, all of the allocators that I’ve looked at should default to a minimum of 16-byte alignment, either by forcing the minimum size of the alignment (like in FMemStackBase::PushBytes) or by using pre-aligned buckets (like in MallocBinned2).

I’ll need to speak to the dev team further to find whether there are any occasions when this assumption over the minimum alignment size could fail. Also, have you made any changes to the default memory allocation settings - like changing the bin size in MallocBinned2.cpp?

In terms of changing FMeshAttributeContainer to use TAlignedHeapAllocator, that could be ok for the control rig code since the attribute data will only persist for the scope of the control rig evaluation. But it wouldn’t be ok for other uses of FMeshAttributeContainer where we have that data persist over multiple frames. So we’d likely need a different type of attribute container if we went down that route. That control rig code used to use a FStackAttributeContainer, but it was changed because control rig stores attributes based on the mesh skeleton, not the compact pose skeleton, and so we ran into crashes where control rig was writing out attributes for bones that didn’t exist in the attribute container.

I’ll follow up once I have more info on this, but it’d be useful to know if you have any customisations as I mentioned.

6

AFAIK we have no change in the MallocBinnes2.cpp

image.png.png
image.png.png1304×251 44.8 KB

7

The dev team are confident that MallocBinned2 will generate 16-byte aligned allocations, so that should be consistent with the MemStackAllocator. The thinking is that this is most likely some kind of memory corruption.

There are a few things you can do so that we can investigate this further. The first would be a full crash dump generated when you hit the check failure. We would also need the symbols along with the dump so that we’re able to look at the state of the memory for the allocation.

Another option is to modify the check that’s failing so that along with printing out PropertyDest, it also prints out the address of Dest, ArrayIndex and Stride. We can then look at the values to see whether they seem valid or not.

Lastly, you can run the project with -stompmalloc passed on the cmd line to enable the stomp malloc allocator. If you run with that, it should generate an exception if anything writes past the end of an allocation. If you hit anything, then that may give a clue as to what the issue is.

8

The idea with running stomp malloc wasn’t to catch the specific occasion when the check fails, but rather to see if it points to something else that’s stomping memory. It might be that it’s happening regularly, but you only see the check fail very occasionally when it just happens to affect that allocation

9

It was a speculation about a possible fix

In the log I have

LogInit: Log: Allocator: Binned2

10

Stomp malloc is not a great option, because we cant reliably repro this.

Thank you for your responses I will keep watch on this and just add more printing to get more info next time