Editor Crash when editing ISM with PerInstanceCustomData (Assertion failed in Scatter)

Development Rendering
, ,
1

Hello,

I am encountering an Editor crash when editing an InstancedStaticMeshComponent (ISM). Could you please provide some insight or a potential fix for this issue?

Reproduction Steps:

  1. Create a new InstancedStaticMeshComponent and set NumCustomDataFloats to 2 (or any non-zero value).
  2. Create a new Material that includes the PerInstanceCustomData node.
  3. Assign this material to the Static Mesh used by the ISM component.
  4. Perform the following sequence: Add instances -> Delete instances (note: the custom data buffer does not appear to shrink) -> Add instances again.

Result: After the final step, a check/assertion within the Scatter() function is triggered, leading to a crash.

[Attachment Removed]

2

重现步骤

  • Create a new InstancedStaticMeshComponent and set NumCustomDataFloats to 2 (or any non-zero value).

  • Create a new Material that includes the PerInstanceCustomData node.

  • Assign this material to the Static Mesh used by the ISM component.

  • Perform the following sequence: Add instances -> Delete instances (note: the custom data buffer does not appear to shrink) -> Add instances again.
    [Attachment Removed]

3

Hi yu lu,

I’ve been assigned this ticket and will attempt to repro and investigate the problem now. I should get back to you soon!

Best regards,

Vitor

[Attachment Removed]

4

Hi yu lu,

Thank you for reporting this issue. I was able to repro the crash in all UE versions starting with 5.4, but the incorrect sizing of the “Custom Data” array seems to happen on even earlier versions.

After some investigation, I noticed that UInstancedStaticMeshComponent::PostEditChangeChainProperty() seems to not be properly handling the removal of elements from the “Instances” array. I am preparing to file an official bug report about this issue, and I’ll get back to you soon with a tracking number for it.

Meanwhile, I believe I was able to implement a quick fix. First, in file “InstancedStaticMesh.cpp”, function UInstancedStaticMeshComponent::PostEditChangeChainProperty(), make the following change:

// BEFORE
else if (PropertyChangedEvent.ChangeType == EPropertyChangeType::ArrayClear)
 
// AFTER
else if (PropertyChangedEvent.ChangeType == EPropertyChangeType::ArrayClear || PerInstanceSMData.IsEmpty())

Now, in function UInstancedStaticMeshComponent::RemoveInstanceInternal() from the same file, make the following change:

// LOCATE THIS BLOCK:
if (!InstanceAlreadyRemoved && PerInstanceSMData.IsValidIndex(InstanceIndex))
{
   (...)
}
 
// ADD RIGHT AFTER IT:
if (InstanceAlreadyRemoved && PerInstanceSMCustomData.Num() > PerInstanceSMData.Num() * NumCustomDataFloats)
{
	if (bUseRemoveAtSwap)
	{
		PerInstanceSMCustomData.RemoveAtSwap(InstanceIndex * NumCustomDataFloats, NumCustomDataFloats, EAllowShrinking::No);
	}
	else
	{
		PerInstanceSMCustomData.RemoveAt(InstanceIndex * NumCustomDataFloats, NumCustomDataFloats);
	}
}

I hope this is helpful. Let me know if the fix above works for you!

Best regards,

Vitor

[Attachment Removed]

5

Hi Vitor,

​

Great to find it finally works!

​

I noticed from the code that InstanceAlreadyRemoved seems to be deliberately set to [TRUE] when running in the Editor.

It appears that the system only flags specific instances for removal, which causes the subsequent logic to skip deleting the corresponding data in PerInstanceSMCustomData.

​

Sincerely,

Lu

[Attachment Removed]

6

Hi yu lu,

Yes, when called from PostEditChangeChainProperty(), that parameter is passed as “true” because the change has supposedly already happened by the interaction of the user with the details panel. But in reality, only the “Instances” array has already been updated at this point, not the “CustomData” array. And function RemoveInstanceInternal() ends up wrongly assuming that neither of these need updating anymore.

In any case, the engine devs should tackle this now (but we cannot give an estimate about when). Here’s the tracking number: UE-361676. This link should become accessible if the devs mark it as public.

All the best,

Vitor

[Attachment Removed]