Weâve managed to find some workarounds for this specific issue, and have quite a bit of extra context that may aid in identifying the actual issue at play. Sorry in advance for the long read!
From the very beginning there seemed to be a disproportionate amount of players getting through on lower end platforms. The following data uses Clicks by platform, and Active Players by platform from the creator portal as there is no Plays by platform.
Prior to our workarounds being implemented, only 2% of clicks were converting into active players on older consoles, which turned into almost 50% afterwards. All platforms saw benefits but this was by far the most dramatic.
This preliminary click â active player data led us to try our own lower spec devices, and we were finally able to consistently replicate an inability to join on both a Nintendo Switch (1) and an Xbox One S.
Note that up until this point weâd only ever received the sporadic âNetwork Connection Lostâ on PC which would usually clear up when retried. On these other devices consistently getting blocked however, the error message was almost exclusively âFailedToConnectâ.
With this consistent failure we were able to start a painstaking binary elimination process after determining that the game loaded fine with no devices in the map at all (verse or otherwise). Removing large chunks of unique devices at a time, uploading a private version, and testing each multiple times on Nintendo Switch eventually led us to three factors that prevented joining on this device:
- Channel Devices
- Verse Tags
- NPC Behavior
Isolating these factors via removing the other ones entirely led to the following results:
- With Channel Devices, we tried one single channel with no links in it, and successfully loaded into the map. We then tried re-making our 7 or so channel devices, but that version gave us FailedToConnect. After this we simply replaced all channels with triggers and this worked.
- With Verse Tags, we were using tags on collectible devices and verse devices in our bossfights. The collectible tags were removed and the devices linked in an array instead, and bosses were reworked to avoid tags, and this let us join again once no Verse Tags were present.
- With NPC Behavior, we removed all NPC Behaviors linked in NPC Character Definitions and reworked our code to let the NPCs still function as expected, which let us join without issue.
With these changes in place, we released an update and saw a roughly 20% increase in players, but were still receiving reports of join issues, which the analytics later backed up. Investigating with an Xbox One S (previously not able to join), led to a crash of Fortnite as a whole after being in the game for a couple minutes. Trying again was seemingly working fine, however another player in the lobby disappeared mid-fight (presumably crashed) and when returning to the lobby, it was clear the session was now in a broken state. Over the course of 5 minutes, 10+ players began to load into the match (visible on the team UI), only to be kicked out instantly within under a second. Attempting to join this lobby on other accounts ourselves yielded the previous âNetwork Connection Lostâ error.
Our initial thought was to disable matchmaking via a round settings device after any player leaves the match so as to prevent any of these broken sessions from being sent new players endlessly, however we got a report that a previously working fine menu was crashing a playerâs game.
Looking into this, we realized we now had a new issue of certain Material Blocks in Verse UI causing crashes on lower end devices and massive 5-10 second lagspikes on PC. We released a hotfix with one material based progress bar removed entirely, and some of the others replaced with simple images instead. This material block fix alone seems to have solved the crashing / broken session issue, as weâd mistakenly removed our disabling of matchmaking code, yet saw a massive increase in players able to get into the game. Looking at the analytics, our matchmaking results to plays ratio is now ~149% whereas previously it was ~50%
We looked into this material block issue that snuck up on us some more and found something very odd. The material block issue is only present when the NPC Character Definitions are not referencing Verse npc_behavior. These two things seem they should be entirely unrelated but a version with the NPC Behaviors linked in the NPC Definitions has no material block issues, and a version with the only change being the links removed from NPCs has the crashing / lagging material block issue again. In case itâs helpful for investigation, 7047-5598-2384 is a private island code in which accessing the âRest Zoneâ menu via the Fishsticks character causes this gigantic lagspike/delay and/or crash. On the same PC, a private version results in a long delay before the menu opens, but in an edit session, the game client fully freezes up until the menu opens. When investigating further, we noticed this lagspike/delay only occurs once every 60 second period after garbage collection. As a series of events, opening the menu causes a huge lagspike/delay, then any subsequent times the menu opens instantly, until garbage collection happens (checked by monitoring the live log file). After garbage collection the lagspike can be received again by opening the menu.
Hereâs a full log file from opening the game, launching the provided private version, opening some menus in the lobby, starting a game and triggering some other material blocks on UI as well, then respawning and closing Fortnite:
FullPrivateCode_FortniteGame.log (4.4 MB)
In this log file, the âUIMat_RZ_Progressâ material block causes a 5 second delay in opening the Rest Zone menu. When this material block was removed, it opened instantly.
Any verse material block shown to the player seems to cause this warning: âLogFort: Warning: [LoadIssue] Synchronously loading file on client while match is in progress and loading screen is not shown: <path/to/material>â, however not every material block shown causes a lagspike or delay. A somewhat complex progress bar, and an incredibly simple icon with an opacity parameter were both causing issues, while some other more complex materials were not.
Some of the other material blocks cause a visible freeze for the client, which is recorded in logs as a âHitch in GameThreadâ right after the previous warnings (search for âM_UI_PermCardEdgeâ).
Hereâs two more focused pieces of log file showing the game hitches caused after these materials are loaded, one in an edit session, and one in what was the live version until recently.
EditSession_RestZoneMenu.log (2.9 KB)
LiveVersion_7-22_PrestigeMenu.txt (4.3 KB)
As seen in these logs, even the prestige menu on the prior live version caused a >1s hitch, however this clearly isnât as bad as the rest zone menuâs >5s hitch, as player counts are way up with just the rest zone material block removed. Our currently released state of the game now has a Hud message device with a UI widget full of all these materials used in material blocks displayed on screen at all times at a 0x0 px size. This seems to prevent these â[LoadIssue]â warnings, as well as any lag from the prestige menu or any other material blocks now.
Let us know if thereâs anything more we can provide or clear up, however this write up has already been a bit⌠extensive. Hope this helps solve whatever may actually be the root cause of all this!
