See Above.
Steps to Reproduce
While studying GAS’ prediction mechanism, I’ve noticed that it’s possible (albeit rare), for GAS to “Leak” predictions such that under certain network conditions or gameplay circumstances, those predictions are never acked/n’acked. This issue is caused when the predicting ASC’s network owner changes during play.
To Reproduce:
- Send a Prediction Key to the Server.
- Change the Network Owner on the Server, either before or on the same frame as receiving the RPC.
- Notice that the prediction never cleans up Client-Side.
There are a number of ways this can manifest:
-
If the RPC arrives at the Server *after* an ownership change, the Server will drop the RPC and therefore never send a response to that client. This is possible since a Client could send the RPC before it knows about the ownership change.
-
If the RPC arrives at the Server *before* an ownership change, but the owner changes that frame, the Client RPC/OwnerOnly property will not be sent to that client at the end of the frame, so they never receive a response.
-
If an Ownership change is received by the Client before the response RPC, or before the OwnerOnly property is receieved, the response will be dropped client-side and never processed.
One potentially trivial way to solve this (untested), would be to have each client locally record all prediction keys that have been dispatched via Server RPC. If a response is received, the key is removed from this list. If the Net owner changes and there are still pending keys in this list, those predictions are rejected and “undone” locally.
There is already a mechanism (non-shipping/debug only) within FScopedPredictionWindow which attempts to catch prediction keys which are used locally but never dispatched to the Server, since ultimately this creates the same problem where predictions may never be undone. The case described above however doesn’t seem to be handled.
This is primarily an issue where an ASC could have multiple owners in it’s lifetime (e.g, a vehicle or weapon actor) - but may also cause problems if a prediction spawns additional actors/objects locally which are then never cleared up (e.g, ASC is destroyed before response received etc).
---
In addition to the above, there seems to be further potential issues where multiple predicting ASC’s can conflict with each other, depending on network conditions or the general net prioritisation of those components.
If many predictions are made in quick succession (or simply before networking can catch up), then depending on the value of CVarStaleKeyBehaviorValue, one ASC’s update can cause another ASC’s predictions to be incorrectly dropped/acked/nacked.
When a key replicates via FReplicatedPredictionKeyItem::OnRep(), the global list of prediction callbacks is iterated over to see how “old” those predictions are relative to the received key (simply by testing the incrementing ID). There seems to be two issues at play here:
- If ASC_1 receives enough prediction respones before ASC_2, then ASC_2’s predictions can be incorrectly flagged as “Stale” and Acked/Nacked/Dropped before ASC_2 has received the real responses.
- Secondly (this compounds with the ownership issue described above) - if we have predictions in that global list which will never receive a response, they too can be flagged as “stale” after enough new responses are received, which may also incorrectly Ack/Nack/Drop them.
A potential way to solve this issue would be to restrict “stale prediction” detection such that a component can only mark it’s own predictions as stale, never those of another component.
Is Epic aware of these potential issues and/or if so, are/would there be any plans to fix this?
Thanks for the report! We know that this issue exists, but don’t have any immediate plans to address it, but will be keeping these issues in mind as we continue to improve and iterate on GAS. We also don’t recommend or think it’s very safe to change NetOwner while exchanging RPCs.
If there aren’t any other questions I am going to close this out on our end.