We ran into a problem where Horde stopped cleaning up build artifacts on our storage backend (a filesystem/NAS backend reached over SMB), and disk usage kept climbing. Investigating, we confirmed artifact retention (keepDays/keepCount) was working correctly at the metadata/API level — expired artifacts disappeared from Horde’s API as expected. The physical blob files on disk, however, were never actually being deleted.
The cause is a bug in the Storage:GC ticker (StorageService.cs), the background job responsible for physically deleting blobs once nothing references them anymore. It’s registered as a single, sequential ticker that only schedules its next run after the previous one fully returns — there is no per-tick timeout or watchdog anywhere in that loop.
Tracing the call chain: TickGcAsync → TickGcForNamespaceAsync → AsyncPipeline.WaitForCompletionAsync(), which does Task.WhenAll over a pool of parallel reachability-check workers and only returns once all of them finish. Each worker eventually calls namespaceInfo.Store.DeleteAsync(…). For a filesystem-backed namespace this resolves to FileObjectStore.DeleteAsync, which turns out to be a fully synchronous method under the hood (File.Delete under a lock, plus a directory-cleanup walk) that accepts a CancellationToken but never actually checks it anywhere in the chain.
If the SMB session to the NAS stalls even briefly — a known failure mode for File.Delete/Directory.Delete against a degraded UNC path — that synchronous call blocks its thread indefinitely, and cannot be interrupted by cancelling a token, because nothing downstream observes one. Since Task.WhenAll won’t complete until every worker returns, one single stuck delete permanently wedges the entire ticker — not just for the affected namespace, but for every namespace on the server, since they all share the same ticker. No exception is ever thrown, so nothing gets logged, and there’s no automatic recovery path. In our case the sweeper had silently been dead for 12+ days across all namespaces before we noticed rising disk usage and started digging. Restarting the Horde server cleared it instantly, which confirms the diagnosis but is obviously not a real fix — the exact same thing can recur any time the NAS has a transient hiccup, with no warning before disk fills up again.
We think the right scope for a fix is the specific call site, not the shared FileObjectStore/underlying storage code (which is used well beyond just Horde’s GC path). A CancellationToken-based timeout won’t work here, since nothing downstream ever reads it — the fix has to race the delete call with a separate timer instead, and on timeout just leave that blob queued for the next sweep rather than blocking forever.
We’d appreciate knowing whether this is a known issue on your side, whether there’s a preferred way of handling it, or whether our proposed fix below looks reasonable before we commit to implementing it.
Proposed fix
// StorageService.cs, CheckReachabilityAsync, around the existing Store.DeleteAsync call
if (storageConfig.EnableGc)
{
Task deleteTask = namespaceInfo.Store.DeleteAsync(objectKey, cancellationToken);
Task winner = await Task.WhenAny(deleteTask, Task.Delay(TimeSpan.FromSeconds(30), cancellationToken));
if (winner != deleteTask)
{
_logger.LogWarning(
"Timed out deleting {NamespaceId} blob {BlobId} from storage (possible stalled I/O); leaving queued for retry next sweep",
namespaceInfo.Id, blobInfoId);
return; // don't mark as deleted; blob stays in the check queue and will be retried
}
try
{
await deleteTask; // guaranteed complete now; surface any real exception
}
// existing missing-file catch block stays as-is
}
[Attachment Removed]