This question was created in reference to: [Strange Behavior with Horde Build Issues Being Opened After They’re [Content removed]
First off, this isn’t a question and is just an update to my original post, as I wanted to share the solution in case anyone else ever encounters the issue. Sincere apologies for the very delayed reply. I had other work priorities come up, then I was out for GDC, then was out again for family reasons. But once back I was finally able to find time to dive in further to debug this issue.
Secondly, thank you for pointing out that not finding the span would cause this behavior, as that gave me a really good starting point!
So after adding a bunch of debug print statements, I was able to confirm that it wasn’t an issue with it not finding the span due to it not finding a matching stream. Instead, it couldn’t find the span at all by that point, even though it was just inserted. I eventually narrowed it down to even after AddSpanAsync inserts the span, immediately trying to find it would come up with 0 spans. At this point it had to be something with the database, so I discussed it with a colleague and she remembered that when standing things up in AWS, the default connection string had &readPreference=secondaryPreferred.
At that point we realized the inconsistency in it working was only if the database insert had been replicated to the secondary databases in time for the read. In the cases where it hadn’t, we get the behavior of it not finding the span and then closing the issue. We’ve since changed the string to primary, and so far it seems to be working!