Crash in FDynamicSkelMeshObjectDataNanite::UpdateBonesRemovedByLOD function when Skeletal Mesh LOD Settings remove the root bone

One of our users was experiencing a crash with their own mod for our game.

After investigation, we’ve found something that seems like an engine bug.

What happened:

User was working on migrating their mod to the new engine version, and started experiencing a crash in the game (no crash in editor)

The crash dump debugging pointed towards a problem with their skeletal mesh configuration, that triggered an engine bug.

The asset was configured to use Nanite, and also had custom LOD settings, that was removing the root bone at LOD 5.

Whenever the actor using this skeletal mesh was spawned, the game crashed

The reason of the crash is accessing array by out of bounds index.

Our findings:

the culprit is in FDynamicSkelMeshObjectDataNanite::UpdateBonesRemovedByLOD function (SkeletalRenderNanite.cpp)

In the second half of the function, there is a for loop:

const FReferenceSkeleton& RefSkeleton = SkinnedAsset->GetRefSkeleton();
for (const FBoneReference& RemovedBone : BonesToRemove)
{
    AllChildrenBones.Reset();
    // can't use FBoneReference::GetMeshPoseIndex() because rendering operates at lower-level (on USkinnedMeshComponent)
    // but this call to FindBoneIndex is probably not so bad since there's typically only the parent bone of a branch in "BonesToRemove"
    const FBoneIndexType BoneIndex = RefSkeleton.FindBoneIndex(RemovedBone.BoneName);
    AllChildrenBones.Add(BoneIndex);
    RefSkeleton.GetRawChildrenIndicesRecursiveCached(BoneIndex, AllChildrenBones);
 
    // first pass to generate component space transforms
    for (int32 ChildIndex = 0; ChildIndex<AllChildrenBones.Num(); ++ChildIndex)
    {
       const FBoneIndexType ChildBoneIndex = AllChildrenBones[ChildIndex];
       const FBoneIndexType ParentIndex = RefSkeleton.GetParentIndex(ChildBoneIndex);
 
       FMatrix44f ParentComponentTransform;
       if (ParentIndex == INDEX_NONE)
       {
          ParentComponentTransform = FMatrix44f::Identity; // root bone transform is always component space
       }
       else if (ChildIndex == 0)
       {
          ParentComponentTransform = static_cast<FMatrix44f>(ComponentSpacePose[ParentIndex].ToMatrixWithScale());
       }
       else
       {
          ParentComponentTransform = PoseBuffer[ParentIndex];
       }
 
       const FMatrix44f RefLocalTransform = static_cast<FMatrix44f>(RefSkeleton.GetRefBonePose()[ChildBoneIndex].ToMatrixWithScale());
       PoseBuffer[ChildBoneIndex] = RefLocalTransform * ParentComponentTransform;
    }
 
    // second pass to make relative to ref pose
    for (const FBoneIndexType ChildBoneIndex : AllChildrenBones)
    {
       PoseBuffer[ChildBoneIndex] = (*RefBasesInvMatrix)[ChildBoneIndex] * PoseBuffer[ChildBoneIndex];
    }
}

Inside this for-loop, there is an access to ComponentSpacePose array by ParentIndex in the “else” block of the “if”.

if (ParentIndex == INDEX_NONE)
       {
          ParentComponentTransform = FMatrix44f::Identity; // root bone transform is always component space
       }
       else if (ChildIndex == 0)
       {
          ParentComponentTransform = static_cast<FMatrix44f>(ComponentSpacePose[ParentIndex].ToMatrixWithScale());
       }

ParentIndex is assigned above it, in this way:

const FBoneIndexType ParentIndex = RefSkeleton.GetParentIndex(ChildBoneIndex);The problem:

When LOD settings are configured to remove the root bone, root bone is added into AllChildrenBones array that is iterated on by the for-loop.

For root bone, the RefSkeleton.GetParentIndex() returns -1.

But FBoneIndexType is defined in BoneIndices.h as:

typedef uint16 FBoneIndexType;So the ParentIndex is uint16, and when assigned a -1 value, the end value saved becomes 65535.

So the initial check:

if (ParentIndex == INDEX_NONE)always fails, as INDEX_NONE is defined as -1, and FBoneIndexType can’t be -1 because it is of unsigned type.

Therefore, the control flow goes into the ELSE block, and accesses ComponentSpacePose array by ParentIndex of 65535, resulting in array index out of bounds crash.

[Attachment Removed]

Steps to Reproduce

Configure the Nanite skeletal mesh Custom LOD settings to remove the skeleton root bone at certain LOD level.

During gameplay, spawn an actor using the configured skeletal mesh, enforcing the LOD level used above.

You should crash.

Can’t tell if it is reproducible in Editor, as we’ve encountered it only in Shipping build.

[Attachment Removed]

This is more of a bug report than a question, we’ve addressed the issue for the user by providing guidance on how to configure the asset LOD settings properly, but the engine code issue still needs to be fixed.

[Attachment Removed]

A couple related bugs on public issues tracker:

UE-291821

UE-254318

[Attachment Removed]

Hello, this code was refactored somewhat in CL#51835229 (fe6284) Fixed artifacts with GPU skin ray tracing LOD when it differs from the main LOD. The problem is the higher ray tracing LOD may reference bones that the raster LOD doesn’t animate. This is the same underyling problem that Nanite faces with LOD0 weights. This change refactors UpdateBonesRemovedByLOD into a common method and applies it to the ray tracing LOD as well.

UE 5.8 CL#52603973 (3a484e)

We’ll do some investigating and see if we can reproduce it in latest.

[Attachment Removed]