Replicating or destroying a subobject marks the entire actor bunch reliable, which may roll back newer property state on the client

Development Programming & Scripting
, , ,
1

I not a native English Speaker, below translated by AI：

Replicating or destroying a subobject marks the entire actor bunch reliable, which can roll back newer property state on the client

Area: Networking / Replication (UActorChannel, FObjectReplicator)
Engine version: UE 4.27 (custom fork) — behavior appears unchanged in UE 5.x
Severity: Client-visible state desync that does not self-heal
Repro rate: Reliably reproducible with packet loss / lag emulation

1. Summary

When an actor channel replicates a newly created subobject, or writes a subobject deletion, the engine sets bReliable = true on the FOutBunch that is being built for that replication pass. That bunch is the single bunch used for the whole actor for that frame — it already contains, or will contain, the property payload of the actor itself and of every other (pre-existing) subobject replicated in the same pass.

The consequence: a snapshot of property values taken at frame N is now carried by a reliable bunch. If that bunch is lost and retransmitted, or held back by send-side queueing, it can be delivered and applied after newer unreliable property bunches have already been applied on the client. The stale values in the reliable bunch then overwrite the newer ones.

Because the server’s FRepLayout shadow state already considers the newer values sent and acknowledged, nothing re-dirties those properties. The client is left in a permanently stale state until the property happens to change again — which, for slowly-changing state, can be seconds, minutes, or never.

2. Relevant engine code

Engine/Source/Runtime/Engine/Private/DataChannel.cpp

New subobject — UActorChannel::ReplicateSubobject():

bool UActorChannel::ReplicateSubobject(UObject* Obj, FOutBunch& Bunch, const FReplicationFlags& RepFlags)
{
    ...
    bool NewSubobject = false;

    if (!ObjectHasReplicator(Obj))
    {
        Bunch.bReliable = true;   // <-- the whole bunch becomes reliable
        NewSubobject = true;
    }
    ...
}

Deleted subobject — UActorChannel::ReplicateActor():

    // Look for deleted subobjects
    ...
    WriteContentBlockForSubObjectDelete(Bunch, ObjID);
    bWroteSomethingImportant = true;
    Bunch.bReliable = true;       // <-- same bunch, same effect

The Bunch passed here is the one FOutBunch Bunch(this, false) created at the top of UActorChannel::ReplicateActor(). All content blocks written during that pass — the actor’s own FRepLayout payload, plus the payload of every other subobject — share it. So reliability is applied at bunch granularity, while the intent seems to be reliability at content-block granularity (only the creation/deletion header truly needs guaranteed delivery).

Receive side — UChannel::ReceivedRawBunch():

    if (Bunch.bReliable && Bunch.ChSequence != Connection->InReliable[ChIndex] + 1)
    {
        // out-of-order reliable bunch -> queued into InRec
    }
    else
    {
        ReceivedNextBunch(Bunch, bOutSkipAck);
    }

Unreliable bunches are never held back by a missing reliable bunch — they are processed immediately. That is what allows the inversion below.

3. Failure timeline

Assume an actor with Health (replicated, changes every frame) and a weapon subobject; at frame N a new subobject S is added.

Time Server Wire Client state after processing
N Builds bunch B1: [create S] + [Actor: Health=100] + [SubObj C: Ammo=30]. bReliable = true (because of S) B1 is lost —
N+1 Bunch B2 (unreliable): [Actor: Health=90] delivered Health = 90
N+2 Bunch B3 (unreliable): [Actor: Health=80], [SubObj C: Ammo=25] delivered Health = 80, Ammo = 25
N+3 NAK for B1 → UChannel::ReceivedNak → B1 retransmitted verbatim (payload is not regenerated) delivered Health = 100, Ammo = 30 ← rollback
N+4… FRepLayout shadow state says Health=80 / Ammo=25 were already sent & acked → nothing is re-dirtied — client stays at 100 / 30

The same inversion happens without packet loss whenever the reliable bunch is delayed relative to later unreliable traffic (send-side queueing / channel saturation / a reliable bunch split into partial bunches that must all arrive before ReceivedNextBunch can assemble it).

Observable symptoms in our project: health / ammo / state-machine enums visibly jumping backwards for one client, then staying wrong; inventory quantity snapping to an older value right at the moment an item subobject is added or removed.

4. Why it does not self-heal

This is the part that makes it more than a one-frame visual glitch:

  1. Property replication is driven by FRepChangedPropertyTracker / FRepLayout comparing against the server-side shadow buffer.
  2. A value that was successfully delivered by an unreliable bunch is considered done. The server has no idea the client later regressed.
  3. The retransmitted reliable bunch is a byte-for-byte replay of an old snapshot; there is no mechanism on the receiving side to reject property data that is older than what has already been applied.

So the only thing that repairs the client is the next genuine change of that property.

5. Minimal repro

  1. Actor with bReplicates = true, one fast-changing replicated property (e.g. Health, updated every tick on the server).
  2. On the server, at some point call AddReplicatedSubObject / return a new subobject from ReplicateSubobjects() (or destroy an existing one).
  3. Apply network emulation on the server: NetEmulation.PktLoss 20 and NetEmulation.PktLag 150.
  4. Log the property value in OnRep / in FRepLayout receive path on the client.
  5. Observe the value stepping backwards exactly on the frames where a subobject is created/destroyed, and staying at the stale value afterwards.

6. Workarounds we have considered

Approach Problem
Add/remove subobjects only on frames where nothing else is dirty Not controllable in practice; the actor’s own properties are dirty almost every frame
Split subobject lifetime changes into their own actor channel / their own pass No supported way to force ReplicateActor to emit a separate bunch for just the content-block header
After the reliable bunch is ACKed, force-dirty everything on the actor and all its subobjects Works, but it is a full property resend on every subobject create/destroy — very expensive for actors with large rep layouts, and it is a guess about when to do it
Client-side “reject old data” guard Requires per-content-block sequencing information the protocol does not carry
Make subobject property payload always reliable Reliable channel head-of-line blocking; unacceptable bandwidth/latency profile for high-frequency state

None of these feel like the intended solution, which is why we would rather ask before shipping a custom engine change.

7. Questions for Epic

  1. Is the bunch-level reliability intentional? Only the content-block header for creation/deletion needs guaranteed delivery. Is there a reason the property payload written in the same pass must also be reliable, beyond “it happens to share the FOutBunch”?
  2. Is it safe to split the bunch? i.e. emit the subobject create/delete content blocks in a dedicated reliable bunch, and keep the actor/subobject property payload in a separate unreliable bunch in the same ReplicateActor() pass. Are there ordering guarantees inside ReceivedNextBunch / FObjectReplicator (content-block → replicator creation → property apply) that this would break? Specifically: can a property block for subobject S be safely processed if the reliable creation block for S arrives in a different bunch that may be delivered later?
  3. Is there an existing mechanism to reject stale property data on receive? Anything like a per-content-block or per-replicator sequence/packet-id stamp we could use, instead of inventing one?
  4. Is force-dirtying the actor + subobjects after the reliable bunch is delivered the recommended mitigation? If so, is there a cheaper engine-supported hook than walking the rep layout and marking everything dirty?
  5. Does UE5 change this? Specifically:
    • the registered subobject list path (net.SubObjects.DefaultUseSubObjectReplicationList) — does it still mark the shared bunch reliable?
    • Iris / ReplicationSystem — is this class of inversion structurally impossible there (given its own attachment/state ordering model)? If Iris is the answer, we would like to know so we can plan the migration rather than patch the legacy path.
  6. Is there an existing bug/CL for this that we can track?

Happy to provide packet captures, LogNetTraffic logs, or a stripped-down repro project on request.