Multiple Runtime Errors with no Callstack since 38.00

Summary

Since version 38.00 (and possibly before), all of our Tycoon games have been receiving a combination of “ErrRuntime_InfiniteLoop”, "ErrRuntime_ComputationLimitExceeded,” and “ErrRuntime_Internal” despite several of them not having been edited in months. Exact timing of onset is unknown as these games were relatively low playercounts at the time, giving us no reason to investigate. Our latest release however is receiving thousands of these errors resulting in broken sessions and players unable to play the game.

Please select what you are reporting on:

Creative

What Type of Bug are you experiencing?

Verse

Steps to Reproduce

The issue seems to manifest sometimes when a player leaves the game. This can be functionally observed as all devices losing functionality in-game right after someone left.
We’ve been unable to determine what exact conditions are necessary for this error to occur

Expected Result

Verse and Creative Devices continue to function as normal when a player leaves the game.
Any errors produce a callstack.

Observed Result

When a player leaves the game there is a chance to cause one of four errors that render Verse AND Creative Devices non-functional (the most prominent being ErrRuntime_InfiniteLoop):

ErrRuntime_InfiniteLoop (>2,200 occurrences)
Verse unrecoverable error: ErrRuntime_InfiniteLoop: The runtime terminated prematurely because Verse code was running in an infinite loop. (Computation timed out - see log for stack trace) (Callstack unavailable)

ErrRuntime_Internal (123 occurrences)
Verse unrecoverable error: ErrRuntime_Internal: An internal runtime error occurred. There is no other information available. (Unable to obtain playspace from device.) (Callstack unavailable)

ErrRuntime_ComputationLimitExceeded (23 occurrences)
Verse unrecoverable error: ErrRuntime_ComputationLimitExceeded: The runtime terminated prematurely because Verse code took too long to execute within a single server tick. Try offloading heavy computation to async contexts. (Script (PersistentLevel_VerseContentScope) has exceeded its maximum running time. Possible infinite loop or excessive computation.) (Callstack unavailable)

ErrRuntime_ComputationLimitExceeded (7 occurrences)
Verse unrecoverable error: ErrRuntime_ComputationLimitExceeded: The runtime terminated prematurely because Verse code took too long to execute within a single server tick. Try offloading heavy computation to async contexts. (Script (RoundLifetimeSimulationEntity_VerseContentScope) has exceeded its maximum running time. Possible infinite loop or excessive computation.) (Callstack unavailable)

Platform(s)

All (Server-side issue)

Island Code

1223-5875-0537

Additional Notes

This can also be observed on our other islands using the same verse framework that were previously running just fine:
3748-4851-5767
3327-2354-3041

We have checked all loops in our code and come to the conclusion that this is most likely a single run of a loop somewhere taking too long to execute rather than a truly infinite loop. Either way the lack of a callstack and simple reproduction makes this almost impossible to debug from our end :frowning:

The status of FORT-1028330 changed to ‘Needs Triage’. We’ve confirmed the issue and it’s waiting to be assigned to someone to fix it.