Mismatch in simulation calculated active emitters and tick 'actual' active emitters

Development Rendering
, ,
1

After upgrading to the latest version of UE, an ensure in NiagaraGPUSystemTick.cpp triggers every now and then: “InSystemInstance->ActiveGPUEmitterCount == InstanceIndex Mismatch in simulation calculated active emitters and tick ‘actual’ active emitters.” Before the ensure, the following warning appears: “GPU Context needed interpolated parameters, but the system instance tick didn’t think we needed to for simulation (%ls). Emiiter will be skipped.”

There seems to have been several changes made around this code recently that acknowledges the issue:

https://github.com/EpicGames/UnrealEngine/commit/55513b2

https://github.com/EpicGames/UnrealEngine/commit/2cb60af

https://github.com/EpicGames/UnrealEngine/commit/5064a28

Is there any fix in sight?

[Attachment Removed]

2

Hi,

Our issue eventually stemmed from dift with the interpolation state of merged emitters, it just took a while to find this out.

I fixed this in “53919084 - Fix interpolated spawn drift between merged script and emitter interpolation mode”, if you know which emitters this is from it might be worthwhile seeing if you are using inheritance or not.

Thanks,

Stu

[Attachment Removed]

3

Thanks Stu! I’ll try cherry-picking your fix.

[Attachment Removed]

4

If that doesn’t work out please let me know.

This has been a tricky one to find as we only had sporadic reports, and it wasn’t until I finally got a solid repro I could track it down, all my other debug code never really gave me useful information.

Thanks,

Stu

[Attachment Removed]

5

Hi Stu,

We’re still seeing this ensure pop up after cherry-picking “53919084 - Fix interpolated spawn drift between merged script and emitter interpolation mode”. It’s quite rare and we don’t have a solid repro unfortunately. Is there any extra information I can provide you that would be useful?

All the best,

Marcus

[Attachment Removed]

6

Hey,

Well that’s unfortunate, I had hoped you would have the same root cause as we did, which was nice and reproducble also.

It might be worthwhile adding something like this in, as it will dump some state to the log.

#if !UE_BUILD_SHIPPING
	if (!ensureMsgf(InSystemInstance->ActiveGPUEmitterCount == InstanceIndex, TEXT("Mismatch in simulation calculated active emitters and tick 'actual' active emitters.")))
	{
		const TSharedPtr<FNiagaraSystemSimulation, ESPMode::ThreadSafe> SystemSim = InSystemInstance->GetSystemSimulation();
		UE_LOGF(LogNiagara, Warning,
			"Niagara GPU active emitter mismatch: System '%ls' Counted(Tick_Concurrent)=%u Built(Init)=%u Delta=%d Mode=%d",
			*GetFullNameSafe(InSystemInstance->GetSystem()),
			InSystemInstance->ActiveGPUEmitterCount, InstanceIndex,
			int32(InSystemInstance->ActiveGPUEmitterCount) - int32(InstanceIndex),
			SystemSim.IsValid() ? int32(SystemSim->GetGPUTickHandlingMode()) : -1
		);
 
		for (const FNiagaraEmitterExecutionIndex& DebugExecIndex : EmitterExecutionOrder)
		{
			const FNiagaraEmitterInstance& DebugEmitter = InSystemInstance->GetEmitters()[DebugExecIndex.EmitterIndex].Get();
			const FNiagaraEmitterInstanceImpl* DebugImpl = DebugEmitter.AsStateful();
			const FNiagaraComputeExecutionContext* DebugContext = DebugEmitter.GetGPUContext();
			UE_LOGF(LogNiagara, Warning,
				"  Emitter[%u] Complete=%d SimTarget=%d Stateful=%d HasContext=%d TickCount=%d HasTicked=%d ExecState=%d Interp=%d",
				DebugExecIndex.EmitterIndex,
				DebugEmitter.IsComplete(),
				int32(DebugEmitter.GetSimTarget()),
				DebugImpl != nullptr,
				DebugContext != nullptr,
				DebugImpl ? DebugImpl->GetTickCount() : -1,
				DebugImpl ? (DebugImpl->HasTicked() ? 1 : 0) : -1,
				int32(DebugEmitter.GetExecutionState()),
				DebugContext ? (DebugContext->HasInterpolationParameters ? 1 : 0) : -1
			);
		}
	}
#else
	ensureMsgf(InSystemInstance->ActiveGPUEmitterCount == InstanceIndex, TEXT("Mismatch in simulation calculated active emitters and tick 'actual' active emitters."));
#endif

I’ll poke around a bit more to see if there’s something obvious in here, I suspect multi-tick / reset pattern that’s causing it which is why it’s so intermitent.

Thanks,

Stu

[Attachment Removed]

7

Hi Marcus,

I’m just back today from summer vacation, thanks for providing the additional context.

I wanted to confirm, does this happen in cooked, editor, or both? And if in cooked do you know if there was anything like a change in scalability before this happened?

Thanks,

Stu

[Attachment Removed]

8

Hi Stu,

I tried adding the logging you proposed and managed to reproduce it once last week. Here’s the output:

Warning: GPU Context needed interpolated parameters, but the system instance tick didn't think we needed to for simulation (<SYSTEM_NAME>:).  Emiiter will be skipped.
Warning: Niagara GPU active emitter mismatch: System 'NiagaraSystem <SYSTEM_NAME>' Counted(Tick_Concurrent)=1 Built(Init)=0 Delta=1 Mode=4
  Emitter[0] Complete=0 SimTarget=1 Stateful=1 HasContext=1 TickCount=1 HasTicked=1 ExecState=0 Interp=1

Does this tell you anything?

All the best,

Marcus

[Attachment Removed]

9

Hi Stu,

We’ve only seen it in cooked client builds, not in editor. None of the reported cases has occurred in connection to changing scalability settings or re-creating render states. It’s always happened mid-gameplay when spawning an effect. There doesn’t seem to be a pattern in which type of Niagara systems trigger the ensure, apart from being GPU simulated.

All the best,

Marcus

[Attachment Removed]

10

Hey Marcus,

Sorry for the delay, I’ve been looking into this and have a change for you to test out, I’ve ran it through our local tests / Fortnite and it has so far been clean.

It’s an option I had been trying to avoid, which is to iterate the emitters while generating the tick, as technically we already have that information. But my theory is that we have an issue when going in / out of solo or a reset in the same frame causing a double tick. Sadly I haven’t been able to make a repro so can’t really verify locally.

I’ve enclosed a patch file that should fix the issue, and hopefully isn’t papering over an issue that you might hit on the dispatch side, I don’t believe so but can’t be 100% certain.

Thanks,

Stu

[Attachment Removed]

11

Thanks Stu!

We’ll give it a try. Since our repro rate is quite low, it might take a while to verify whether it fixes the issue. I’m also away on vacation for 3 weeks starting next week.

All the best,

Marcus

[Attachment Removed]

12

Thanks, enjoy your vacation, and hopefully when your back there have been no crashes :slight_smile:

Thanks,

Stu

[Attachment Removed]

13

I can confirm this seems to have fixed it. No ensure conditions failed in the past 4 weeks. Thanks!

[Attachment Removed]

14

Great thanks for letting me know, happy to hear we have finally nailed this one!

[Attachment Removed]