Hey there Abhishek,
I’ve spoken with the team a bit on this issue, and have looked internally at some of our job history.
This issue has come up in the past, and has not been an issue in that it doesn’t appear to be a fatal outcome. We have some hypothesis (thanks for the sanity check [mention removed] ):
- An upload is retried? This call is not idempotent
- There’s truly a duplicated blob due to content itself
- There’s a bug in the step-output handling
- This outcome can happen by-design but just not handled
We feel as though it could be 1 and/or 4 as the most plausible outcome here.
One thing you could try here is the following divergence within AddBlobAsync (StorageService.cs):
`internal async Task AddBlobAsync(NamespaceId namespaceId, BlobLocator locator, IReadOnlyCollection imports, List? exports = null, CancellationToken cancellationToken = default)
{
List importIds = await FindOrAddShadowBlobsAsync(namespaceId, imports, cancellationToken);
FilterDefinition insertFilter =
Builders.Filter.Eq(x => x.NamespaceId, namespaceId) &
Builders.Filter.Eq(x => x.Path, locator.ToString()) &
Builders.Filter.Eq(x => x.Shadow, true);
UpdateDefinition insertUpdate = Builders.Update
.Unset(x => x.Shadow);
if (importIds.Count > 0)
{
insertUpdate = insertUpdate.Set(x => x.Imports, importIds);
}
if (exports != null && exports.Count > 0)
{
insertUpdate = insertUpdate.Set(x => x.Aliases, exports);
}
// JULIAN_DIVERGENCE_START
BlobInfo? blobInfo;
try
{
blobInfo = await _blobCollection.FindOneAndUpdateAsync(insertFilter, insertUpdate, new FindOneAndUpdateOptions<BlobInfo, BlobInfo> { IsUpsert = true, ReturnDocument = ReturnDocument.After }, cancellationToken);
}
catch (MongoWriteException ex) when (ex.WriteError.Category == ServerErrorCategory.DuplicateKey)
{
_logger.LogWarning(“Race condition on upsert for blob {Path}. Retrying as read-only lookup.”, locator.ToString());
blobInfo = await _blobCollection.Find(insertFilter).FirstOrDefaultAsync(cancellationToken);
}
// JULIAN_DIVERGENCE_END
_logger.LogDebug(“Created blob {BlobId} at {Path} ({NumImports} imports)”, blobInfo.Id, blobInfo.Path, imports.Count);
return blobInfo;
}`Since you have a very consistent occurrence of this, if we see nominal behaviour on your end this could be a potential fix we could adopt on our end. The aim of the above code is more or less to try and add, if not, we perform a find for the given filter and return the object.
Kind regards,
Julian