We are investigating increasingly frequent artifact retrieval issues with our Horde installation and would appreciate Epic’s input.
Over the last few weeks we have seen:
- Artifact retrieval timeouts becoming more frequent
- Occasional retrieved data whose size differs from the expected artifact/manifest data
- Very slow artifact retrieval
- Horde UI operations that need to retrieve storage-backed data becoming slow at the same time
- No obvious CPU or memory saturation on the Horde Server
- SMB throughput that does not appear saturated
- Occasional SMB/file access latency spikes
We traced the artifact read path down to:
BatchBlobReader -> BundleHandle -> FileStorageBackend -> FileObjectStore -> MemoryMappedFileCache
One thing caught our attention.
FileObjectStore.ReadAsync() is effectively synchronous and eventually calls MemoryMappedFileCache.Read(). MemoryMappedFileCache.Read() holds a global lock while FindOrAddMappedFile() runs. On a cache miss, FindOrAddMappedFile() performs filesystem operations such as checking FileInfo.Exists, FileInfo.Length, possible cache eviction, and creation of a MemoryMappedFile / view accessor.
Because our files are on an SMB UNC path, it appears that remote filesystem operations can occur while that global lock is held.
We also confirmed that FileObjectStoreFactory is registered as a singleton and owns a single MemoryMappedFileCache shared by the FileObjectStore instances it creates.
The default MemoryMappedFileCache limits are:
1 GB maximum mapped size
128 mapped files
Our current hypothesis is that occasional SMB latency could be amplified by this shared lock. If opening or mapping an uncached blob takes hundreds of milliseconds, other concurrent artifact reads using the same cache could wait behind it, temporarily serializing reads.
This may also explain why UI operations that need storage-backed data slow down at the same time as artifact retrieval.
We have not confirmed this as the root cause yet. Our next planned test is to instrument MemoryMappedFileCache.Read() to separately measure:
- time waiting to acquire the lock
- time spent in FindOrAddMappedFile()
- Is using a FileSystem backend over SMB with roughly 20 ms RTT an expected/recommended Horde topology?
- Does Epic use remote SMB/NAS storage for Horde artifacts internally, or is low-latency/object storage generally expected?
- Could MemoryMappedFileCache’s shared lock become a bottleneck with higher-latency filesystem storage?
- Are the default 1 GB / 128 mapped file limits expected to work well for large artifact workloads?
- Would increasing those limits be a reasonable diagnostic test?
We are mainly trying to determine whether this is a NAS/network issue, a storage topology issue, or SMB latency being amplified by Horde’s FileSystem backend implementation.
[Attachment Removed]
