All Island sessions keep crashing after 3,4,5 rounds (AFTER NEW SEASON)

General Issues and Bug Reporting
, , , , ,
1

Please select what you are reporting on:

Creative

What Type of Bug are you experiencing?

Stability

Summary

BIG UPDATE: We were fooled by the timing of things by trying quitting and rejoining.
The crash seems to happen all the time after 3-4-5 rounds without the need of connecting or disconnecting. This has been happening since new season update seems.
We tried to roll back to old “working” versions of previous season on which we did also lives on twitch and were originally working for an undefined number of rounds but after 3-4-5 rounds session crash for all players without the need to quit or rejoin.

LogVerse: log_waiting_for_more_players_device: Waiting if there are enough players for the round to start.
LogVerse: log_waiting_for_more_players_device: Starting round.
LogVerse: log_prop_hunt_device: Round started.
LogVerse: log_prop_hunt_device: Setting 1 hunter agent(s).
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: Setting 1 prop agent(s).
LogVerse: log_team: Setting a new prop agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: Teleporting hunter agent(s).
LogVerse: log_prop_hunt_device: Starting prop(s) game logic.
LogVerse: log_round_timer_device: Starting the round timer.
LogVerse: log_round_timer_device: Round timer running.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_prop_hunt_device: A player joined the game.
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_prop_hunt_device: Hunter agent eliminated a prop agent.
LogVerse: log_prop_hunt_device: Prop agent eliminated.
LogVerse: log_team: Prop agent eliminated.
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_team: 0 agent(s) on team remaining.
LogVerse: log_team: No agents on team remaining. Ending the round.
LogVerse: log_prop_hunt_device: Round ending.
LogVerse: log_prop_hunt_device: Round ended.
LogVerse: log_prop_hunt_device: Round loaded.
LogVerse: log_prop_hunt_device: Setting up teams.
LogVerse: log_waiting_for_more_players_device: Waiting if there are enough players for the round to start.
LogVerse: log_waiting_for_more_players_device: Starting round.
LogVerse: log_prop_hunt_device: Round started.
LogVerse: log_prop_hunt_device: Setting 1 hunter agent(s).
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: Setting 2 prop agent(s).
LogVerse: log_team: Setting a new prop agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_team: Setting a new prop agent.
LogVerse: log_team: Adding agent to team.
LogHttp: Warning: 00000961F6DF1C00: request failed, libcurl error: 28 (Timeout was reached)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 0 (  Trying 10.99.35.41:443...)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 1 (ipv4 connect timeout after 14981ms, move on!)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 2 (  Trying 10.99.34.52:443...)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 3 (ipv4 connect timeout after 7490ms, move on!)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 4 (  Trying 10.99.34.230:443...)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 5 (ipv4 connect timeout after 7488ms, move on!)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 6 (Failed to connect to horde.devtools-dev.epicgames.com port 443 after 30000 ms: Timeout was reached)
LogHttp: Warning: 00000961F6DF1C00: libcurl info message cache 7 (Closing connection)
LogHttp: Warning: 00000961F6DF0E00: request failed, libcurl error: 28 (Timeout was reached)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 0 (  Trying 10.99.51.85:443...)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 1 (ipv4 connect timeout after 14972ms, move on!)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 2 (  Trying 10.99.50.4:443...)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 3 (ipv4 connect timeout after 7484ms, move on!)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 4 (  Trying 10.99.50.235:443...)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 5 (ipv4 connect timeout after 7485ms, move on!)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 6 (Failed to connect to horde.devtools.epicgames.com port 443 after 30000 ms: Timeout was reached)
LogHttp: Warning: 00000961F6DF0E00: libcurl info message cache 7 (Closing connection)
LogVerse: log_prop_hunt_device: Teleporting hunter agent(s).
LogVerse: log_prop_hunt_device: Starting prop(s) game logic.
LogVerse: log_round_timer_device: Starting the round timer.
LogVerse: log_round_timer_device: Round timer running.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_prop_hunt_device: Hunter agent eliminated a prop agent.
LogVerse: log_prop_hunt_device: Prop agent eliminated.
LogVerse: log_team: Prop agent eliminated.
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_team: 1 agent(s) on team remaining.
LogVerse: log_prop_hunt_device: Hunter agent eliminated a prop agent.
LogVerse: log_prop_hunt_device: Prop agent eliminated.
LogVerse: log_team: Prop agent eliminated.
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_team: 0 agent(s) on team remaining.
LogVerse: log_team: No agents on team remaining. Ending the round.
LogVerse: log_prop_hunt_device: Round ending.
LogVerse: log_prop_hunt_device: Round ended.
LogVerse: log_prop_hunt_device: Round loaded.
LogVerse: log_prop_hunt_device: Setting up teams.
LogVerse: log_waiting_for_more_players_device: Waiting if there are enough players for the round to start.
LogVerse: log_waiting_for_more_players_device: Starting round.
LogVerse: log_prop_hunt_device: Round started.
LogVerse: log_prop_hunt_device: Setting 1 hunter agent(s).
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: Setting 1 prop agent(s).
LogVerse: log_team: Setting a new prop agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: A player joined the game.
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogNet: Warning: UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.01, Real: 30.01, Good: 30.01, DriverTime: 755.91, Threshold: 30.00, [UNetConnection] RemoteAddr: 4.185.92.1:15039, Name: IpConnection_3, Driver: Name:IpNetDriver_3 Def:BeaconNetDriver IpNetDriver_3, IsServer: NO, PC: NULL, Owner: ValkyrieBeaconClient_3, UniqueId: MCP:2b19bbc3bece460887df39f0a537747e
LogNet: Error: UEngine::BroadcastNetworkFailure: FailureType = ConnectionTimeout, ErrorString = UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.01, Real: 30.01, Good: 30.01, DriverTime: 755.91, Threshold: 30.00, [UNetConnection] RemoteAddr: 4.185.92.1:15039, Name: IpConnection_3, Driver: Name:IpNetDriver_3 Def:BeaconNetDriver IpNetDriver_3, IsServer: NO, PC: NULL, Owner: ValkyrieBeaconClient_3, UniqueId: MCP:2b19bbc3bece460887df39f0a537747e, Driver = Name:IpNetDriver_3 Def:BeaconNetDriver IpNetDriver_3
LogValkyrieRequestManagerEditor: Warning: Encountered Network Error!
LogValkyrieRequestManagerEditor: Error: Failed to handle connection attempt (Network error occurred when connecting to a server after matchmaking completed. Error: UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.01, Real: 30.01, Good: 30.01, DriverTime: 755.91, Threshold: 30.00, [UNetConnection] RemoteAddr: 4.185.92.1:15039, Name: IpConnection_3, Driver: Name:IpNetDriver_3 Def:BeaconNetDriver IpNetDriver_3, IsServer: NO, PC: NULL, Owner: ValkyrieBeaconClient_3, UniqueId: MCP:2b19bbc3bece460887df39f0a537747e)
LogValkyrieSummary: Destroying Valkyrie Beacon

Steps to Reproduce

1.Enter a game with at least 2 players
2.Play until round 3,4 or 5
3. Wait for the crash

Expected Result

Session crashes for all players, resulting in a some sort of freeze for all players (probably not receiving updates anymore from the server that crashed) and then sent to lobby with “Connection Lost” various network related message

Observed Result

Session crashes for all players, resulting in a some sort of freeze for all players (probably notreceiving updates anymore from the server) and then sent to lobby with “Connection Lost” / various network related message

Platform(s)

PC, PS4

Island Code

3548-5795-5105

Additional Notes

We tried to keep the map as lightweight as possible (currently at 70.000), tried to remove custom models and avoid non native functions and everything in our power but problem keep persisting. (We are desperate)
The map is based of the standard prop hunt verse template given by Epic Games and it has been untouched in terms of code except two hud_message_device added on round ending and minor changes that was previously working.

We noticed problems started on season update.

2

Addional test, this time the lobby crashed without anyone quitting.
We just started a new game in 3 players

As soon as the hunter were released the session crashed

LogValkyrie: ProjectSizeRequest::GetProjectCookedSize for a00962db-4e66-3f9a-f754-c7822d9ca24d
LogValkyrie: ProjectSizeRequest::GetDependencyDownloadSizes for a00962db-4e66-3f9a-f754-c7822d9ca24d
LogValkyrie: ProjectSizeRequest::Completed for a00962db-4e66-3f9a-f754-c7822d9ca24d
LogValkyrieSummary: Server Summary - Successfully activated content on all platforms
LogVerse: log_prop_hunt_device: Round loaded.
LogVerse: log_prop_hunt_device: Setting up teams.
LogVerse: log_waiting_for_more_players_device: Waiting if there are enough players for the round to start.
LogVerse: log_waiting_for_more_players_device: Starting round.
LogVerse: log_prop_hunt_device: Round started.
LogVerse: log_prop_hunt_device: Setting 1 hunter agent(s).
LogVerse: log_team: Setting a new hunter agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: Setting 2 prop agent(s).
LogVerse: log_team: Setting a new prop agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_team: Setting a new prop agent.
LogVerse: log_team: Adding agent to team.
LogVerse: log_prop_hunt_device: Teleporting hunter agent(s).
LogVerse: log_prop_hunt_device: Starting prop(s) game logic.
LogVerse: log_round_timer_device: Starting the round timer.
LogVerse: log_round_timer_device: Round timer running.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogVerse: log_round_timer_device: Creating new round time text block.
LogVerse: log_round_timer_device: Adding round timer UI to a player.
LogNet: Warning: UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.01, Real: 30.01, Good: 30.01, DriverTime: 618.34, Threshold: 30.00, [UNetConnection] RemoteAddr: 4.185.92.1:15059, Name: IpConnection_5, Driver: Name:IpNetDriver_5 Def:BeaconNetDriver IpNetDriver_5, IsServer: NO, PC: NULL, Owner: ValkyrieBeaconClient_5, UniqueId: MCP:2b19bbc3bece460887df39f0a537747e
LogNet: Error: UEngine::BroadcastNetworkFailure: FailureType = ConnectionTimeout, ErrorString = UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.01, Real: 30.01, Good: 30.01, DriverTime: 618.34, Threshold: 30.00, [UNetConnection] RemoteAddr: 4.185.92.1:15059, Name: IpConnection_5, Driver: Name:IpNetDriver_5 Def:BeaconNetDriver IpNetDriver_5, IsServer: NO, PC: NULL, Owner: ValkyrieBeaconClient_5, UniqueId: MCP:2b19bbc3bece460887df39f0a537747e, Driver = Name:IpNetDriver_5 Def:BeaconNetDriver IpNetDriver_5
LogValkyrieRequestManagerEditor: Warning: Encountered Network Error!
LogValkyrieRequestManagerEditor: Error: Failed to handle connection attempt (Network error occurred when connecting to a server after matchmaking completed. Error: UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.01, Real: 30.01, Good: 30.01, DriverTime: 618.34, Threshold: 30.00, [UNetConnection] RemoteAddr: 4.185.92.1:15059, Name: IpConnection_5, Driver: Name:IpNetDriver_5 Def:BeaconNetDriver IpNetDriver_5, IsServer: NO, PC: NULL, Owner: ValkyrieBeaconClient_5, UniqueId: MCP:2b19bbc3bece460887df39f0a537747e)
LogValkyrieSummary: Destroying Valkyrie Beacon

We cannot really determine what causes it

Another time it just went like this on round reset.

3

If you are handling players in any way then you must handle them throughout their entire lifecycle in the game.

I’d recommend looking into utilizing OnPlayerRemoved(…) to clean up anything that would require additional attention.

This is a common mistake if using Verse UI’s.

4

OnPlayerAdded and Removed are present as function subcribed by playerspace just calls functions to remove or add agents in the two teams Props and Hunters teams by initializing them with the right classSelector device but besides that there is nothing related to player persisting in game session.
At the round start the players are splitted and that has two local scoped arrays that I expect to be destroyed by garbage collector as soon as the function to SetUpTeams() end.

Here is the template for every verse file just in case

Moreover verse instantiations should automatically reset/cleanup on round restarting if I don’t get it wrong (besides weak_maps that i’m NOT currently using?)

Another thing I’m noticing is that the map has been placed really high in terms of altitude (or Z if you want). Something like 20000 units above the 0 level.
Might this impact game performance? Starting much above what should be the standard “sky” of the game might impact this crashes? I noticed that I cannot drive flying vehicles at that altitude and constatly tries to push them back at lower altitudes resulting in a “magnet like behaviour” that attaches the flying vehicle to the below terrain.

I’m trying to traslate the whole map back to a 0 like altitute and see if it gets better

EDIT: Bringing the map near the origin did not fix the problem but maybe speedup loading time??

5

BIG UPDATE: We were fooled by the timing of things by trying quitting and rejoining.
The crash seems to happen all the time after 3-4 rounds without the need of connecting or disconnecting. This has been happening since new season update seems.
We tried to roll back to old “working” versions on which we did also lives on twitch and were originally working for an undetermined number of rounds but after 3-4-5 rounds session crash constantly.

immagine
immagine1729×1008 39.4 KB

I add some fortniteGame log if that might be useful to any dev.
It should contain the last matches played, as soon as the game crashed I hit alt+f4 so the last log should be representative of game state before I got the Network Connection Lost error
FortniteGame.log (15.6 MB)

6

Here’s an update that might be useful to someone:

We spent four days debugging our map. We started by stripping it down to its basic functionalities to see if the error persisted. After confirming it didn’t, we gradually rolled back and removed elements piece by piece, publishing a private version each time. It was an intense debugging session, probably the most challenging I’ve experienced in my career as a software engineer.

Working with creative maps is incredibly frustrating. You have no control over backend failures and there’s a lack of clear logs.

We evaluated every possible cause: cinematics, devices, audio buttons, custom textures, and verse code. We deleted them one by one to find what was causing the server crash.

It turns out the turrets were the problem. We had six sentries around the map that randomly crashed the game after several rounds or when a player disconnected and reconnected. Removing the automated turrets stopped the server crashes.

It seems something builds up in memory across rounds that the garbage collector can’t delete, leading to a memory overflow or maybe just a problem with resource resetting/reloading (just a guess, since I’m not a Fn developer).

I can’t say if the turrets were broken by the new season, but previous recorded live sessions didn’t show this behavior and all turrets were present: YouTube Video. (The video was uploaded later but was made during the previous season.)