Horde - cleanup deleting only refs not files

Development Pipeline & Plugins
, ,
1

Hi,

we are using Horde for CI/CD stuff. We noticed, that Horde has system for automatic artifacts cleanup. But im not sure, if its behave correctly, or am i misunderstanding something.

We have defined artifactTypes property for each artifact type (packaged-build,staged-build,step-output,step-saved,step-testdata,step-trace,ugs-pcb) to be keepCount 3 or 5 last builds. Artifact types are defined in *.stream.json for our project in plugins->build->projects->streams->artifactTypes

But when i keep tracking of horde artifacts storage, there are much more data than last 3-5 builds. When i look into horde server logs, there are informations about expiring and removing artifact refs, but nothing about deletion. When i use swagger to explore horde rest api (/api/v2/artifacts) to list horde artefacts, this output looks like correctly deleted. But only those references, not files on horde server.

Is horde able to remove files too? Or it only cleanups reference, and files needs to be cleaned other way? I thought about some delay between marking result as expired and deletion, but we have even two months old artefacts here.

Any hint what am i doing wrong?

2

Hey there Lubos,

Can you provide some of the following details:

  • Horde Server version
  • Your configuration of these artifact type (that is, what’s the storage backend for these namespaces)

Kind regards,

Julian

3

Hi Julian,

sorry my mistake, i thought i added those information in attachment but forgot to attach it.

Horde server version is 5.5.1-0 (screenshot of all versions in attachment)

And for configuration of artifact types (full configuration json in attachment. Config was exported from horde server via rest api)

"artifactTypes": [ { "name": "packaged-build", "type": "packaged-build", "keepCount": 3, "namespaceId": "horde-artifacts" }, { "name": "staged-build", "type": "staged-build", "keepCount": 3, "namespaceId": "horde-artifacts" }, { "name": "step-output", "type": "step-output", "keepCount": 3, "namespaceId": "horde-artifacts" }, { "name": "step-saved", "type": "step-saved", "keepCount": 3, "namespaceId": "horde-artifacts" }, { "name": "step-testdata", "type": "step-testdata", "keepCount": 3, "namespaceId": "horde-artifacts" }, { "name": "step-trace", "type": "step-trace", "keepCount": 3, "namespaceId": "horde-artifacts" }, { "name": "ugs-pcb", "type": "ugs-pcb", "keepCount": 5, "namespaceId": "horde-artifacts" } ]And horde-artifacts namespace is defined

{ "id": "horde-artifacts", "backend": "default-backend", "prefix": "Artifacts/", "gcFrequencyHrs": 0.1, "gcDelayHrs": 6, "enableAliases": false, "acl": {} },Where default-backed is defined

{ "id": "default-backend", "base": "", "secondary": "", "type": "FileSystem", "baseDir": "Storage" },Thanks and regards,

Lubos

4

Hey there Lubos,

Good stuff - thanks for this. So you’re 5.5.1 version had quite a few bugs in the storage service; I’d strongly recommend upgrading to 5.5.4 as it will contain a large degree of performance improvements and bug fixes around garbage collection.

Julian

5

Hey, we are looking into very similar topic this week. Our artifacts storage (5.5.4) got full and clean up was not working. Going over logs showed that there seem to be two phases, one is ref clean up which happens based on the schedule and then later garbage collection happens which actually deleted the files. In our case, artifacts storage garbage collection was giving exception. We are not sure how it happened, whether something was really deleted manually on the machine but this exception halts garbage collection for the whole storage instead of just skipping a single blob which failed. This causes our artifacts to just grow up without any clean up. Maybe check if that could be also your case. We had some power outages / hard server resets recently, unclear if related. We are not sure how to get out of this state, so will be redeploying the Horde.

[00:00:12 inf] Running garbage collection for namespace horde-artifacts... [00:00:12 inf] Garbage collection queue for namespace horde-artifacts (storage:horde-artifacts:check) has 50517 entries [00:00:12 dbg] Deleting horde-artifacts blob 680a78b0e3b95d3b81e39c3d, key: ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/8bf116012d8243fc823394c3dbcb184c_7.blob (0 imports) [00:00:12 dbg] Deleting horde-artifacts blob 680a78ade3b95d3b81e39b87, key: ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/8bf116012d8243fc823394c3dbcb184c_6.blob (0 imports) [00:00:12 dbg] Deleting horde-artifacts blob 680a78b0e3b95d3b81e39c0f, key: ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/6c9f0eac54a0478fb8937e94bdaa3594_3.blob (0 imports) [00:00:12 dbg] Deleting horde-artifacts blob 680a78aee3b95d3b81e39bab, key: ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/6c9f0eac54a0478fb8937e94bdaa3594_1.blob (0 imports) [00:00:12 dbg] Deleting horde-artifacts blob 680a78b0e3b95d3b81e39c63, key: ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/6c9f0eac54a0478fb8937e94bdaa3594_2.blob (0 imports) [00:00:12 err] Exception while running garbage collection: Could not find a part of the path '/app/Data/Storage/artifacts2/ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/8bf116012d8243fc823394c3dbcb184c_7.blob'. System.IO.DirectoryNotFoundException: Could not find a part of the path '/app/Data/Storage/artifacts2/ugs-pcb/ftw-release-0.3.0.0/119528/editor/680a78a8fe720458f0c72627/8bf116012d8243fc823394c3dbcb184c_7.blob'. at System.IO.FileSystem.DeleteFile(String fullPath) at EpicGames.Core.MemoryMappedFileCache.Delete(FileReference file) in /app/Source/Programs/Shared/EpicGames.Core/MemoryMappedFileCache.cs:line 235 at EpicGames.Horde.Storage.ObjectStores.PrefixedObjectStore.DeleteAsync(ObjectKey locator, CancellationToken cancellationToken) in /app/Source/Programs/Shared/EpicGames.Horde/Storage/ObjectStores/PrefixedObjectStore.cs:line 38 at HordeServer.Storage.StorageService.CheckReachabilityAsync(NamespaceInfo namespaceInfo, SortedSetEntry1 entry, ObjectId lastImportBlobInfoId, GcSweepState state, StorageConfig storageConfig, AsyncEvent queueChangeEvent, CancellationToken cancellationToken) in /app> at EpicGames.Core.AsyncPipelineExtensions.ProcessItemsAsync[T](ChannelReader1 reader, Func3 taskFunc, CancellationToken cancellationToken) in /app/Source/Programs/Shared/EpicGames.Core/AsyncPipeline.cs:line 202
at EpicGames.Core.AsyncPipeline.RunGuardedAsync(Func2 taskFunc) in /app/Source/Programs/Shared/EpicGames.Core/AsyncPipeline.cs:line 97 at EpicGames.Core.AsyncPipeline.WaitForCompletionAsync() in /app/Source/Programs/Shared/EpicGames.Core/AsyncPipeline.cs:line 125 at HordeServer.Storage.StorageService.TickGcForNamespaceAsync(NamespaceInfo namespaceInfo, ObjectId lastImportBlobInfoId, DateTime utcNow, CancellationToken cancellationToken) in /app/Source/Programs/Horde/Plugins/Storage/HordeServer.Storage/Storage/StorageService.cs> at HordeServer.Storage.StorageService.TickGcForNamespaceAsync(NamespaceInfo namespaceInfo, ObjectId lastImportBlobInfoId, DateTime utcNow, CancellationToken cancellationToken) in /app/Source/Programs/Horde/Plugins/Storage/HordeServer.Storage/Storage/StorageService.cs> at HordeServer.Storage.StorageService.TickGcAsync(CancellationToken cancellationToken) in /app/Source/Programs/Horde/Plugins/Storage/HordeServer.Storage/Storage/StorageService.cs:line 1462 [00:00:12 inf] Running garbage collection for namespace default... [00:00:12 inf] Garbage collection queue for namespace default (storage:default:check) has 0 entries [00:00:12 inf] Garbage collection complete for namespace default [00:00:12 inf] Finished garbage collection for namespace default in 0.0010861s (0 removed) [00:00:12 inf] Running garbage collection for namespace horde-logs... [00:00:12 inf] Garbage collection queue for namespace horde-logs (storage:horde-logs:check) has 0 entries [00:00:12 inf] Garbage collection complete for namespace horde-logs [00:00:12 inf] Finished garbage collection for namespace horde-logs in 0.0008352s (0 removed) [00:00:12 inf] Running garbage collection for namespace horde-perforce... [00:00:12 inf] Garbage collection queue for namespace horde-perforce (storage:horde-perforce:check) has 0 entries [00:00:12 inf] Garbage collection complete for namespace horde-perforce [00:00:12 inf] Finished garbage collection for namespace horde-perforce in 0.0010788s (0 removed) [00:00:12 inf] Running garbage collection for namespace horde-tools... [00:00:12 inf] Garbage collection queue for namespace horde-tools (storage:horde-tools:check) has 0 entries [00:00:12 inf] Garbage collection complete for namespace horde-tools [00:00:12 inf] Finished garbage collection for namespace horde-tools in 0.0009697s (0 removed) [00:00:12 inf] Running garbage collection for namespace memory... [00:00:12 inf] Garbage collection queue for namespace memory (storage:memory:check) has 0 entries [00:00:12 inf] Garbage collection complete for namespace memory [00:00:12 inf] Finished garbage collection for namespace memory in 0.0009456s (0 removed)

6

Hey there Ondrej,

Thanks for the response here; I’ve created a separate ticket for this. We should fail a bit more gracefully in this scenario, and cleanup dangling refs so that subsequent GCs can occur.

Julian

7

Thank you!