Horde ManagedWorkspace can retain incorrect file content when Repository.dat metadata matches but physical MD5 differs

We found a ManagedWorkspace consistency issue. After a completed sync to //xxx/demo, the Perforce client stream, have table, and Repository.dat all identified a file as the Demo revision, but the physical file contained bytes from //xxx/main. Later job syncs did not detect or repair the mismatch.

Both streams map this file to the same workspace path:

//xxx/main/XXX/Content/GameData/Effects/Animal_V2/Materials/MI_AnimalFX_PawTrail.uasset

//xxx/demo/XXX/Content/GameData/Effects/Animal_V2/Materials/MI_AnimalFX_PawTrail.uasset

P4 metadata at the affected changelist:

Main: binary+l, revision 4, size 11696

MD5: 777395DA8EC781D3C47EB22714D4F5AC

Demo: binary+l, revision 1, size 11696

MD5: F4D60FF298F4CE610574FC01424C5AC7

Affected client:

Root: D:\HordeAgent\Sandbox\xxx-Full\Sync

Stream: //xxx/demo

The have table reported:

//xxx/demo/XXX/Content/GameData/Effects/Animal_V2/Materials/MI_AnimalFX_PawTrail.uasset#1

`p4 have` reported Demo revision 1. The `Repository.dat` workspace entry also contained the Demo `ContentId`:

Length: 11696

ReadOnly: true

Digest: F4D60FF298F4CE610574FC01424C5AC7

Type: binary+l

However, hashing the physical workspace file returned the Main digest:

777395DA8EC781D3C47EB22714D4F5AC

Therefore:

P4 have table: Demo revision

Repository.dat: Demo ContentId

Physical file: Main bytes

The Main and Demo versions have equal length but different content.

A Demo cook consumed Main asset content and referenced other assets available only in Main.

We observed a similar have-table/disk inconsistency with another file on a different ManagedWorkspace.

`FileContentId` correctly distinguishes these revisions using the P4 digest and file type. The problem appears to be physical integrity validation.

`WorkspaceDirectoryInfo.RefreshAsync()`, `CachedFileInfo.CheckIntegrity()`, and `repaircache` validate existence, length, `LastWriteTimeUtc`, and readonly state, but do not recalculate the physical MD5.

The two revisions have identical size. `Repository.dat` also contained matching timestamp and attributes, so metadata validation succeeded even though the bytes did not match its `ContentId`. Once wrong bytes become associated with the expected `ContentId`, subsequent syncs can continue trusting them.

We do not know what originally introduced the wrong bytes. Possible causes include cache contamination, transaction recovery, concurrent/external writes, or an unexpected P4 result. The confirmed issue is that ManagedWorkspace cannot detect or recover from this state after it exists.

  1. Is this a known ManagedWorkspace issue or limitation?
  2. Is one ManagedWorkspace expected to safely switch sequentially between streams?
  3. Is there an existing supported full content-verification or recovery mechanism?
  4. Should `repaircache` verify physical content against `FileContentId`?
  5. Would Epic recommend validation during `Sync -> Cache`, `Cache -> Sync`, or after P4 download?
  6. How should transformed text and symlink types be handled when depot MD5 may differ from local bytes?
  7. Is PerforceMaterializer or a separate workspace identifier per stream the recommended mitigation?

Our proposed low-impact fix is opt-in binary verification (`perforce repaircache -VerifyContent`) plus MD5 validation for binary files at Sync/Cache boundaries. A mismatch would discard the untrusted entry and force a P4 download. Normal metadata-only scans and the `Repository.dat` format would remain unchanged.

Is there a better supported approach or intended integration point?

[Attachment Removed]

Thank you for bringing this to our attention, and apologies for any impact it may have caused you and the delay in getting back to you.

I have been discussing this with the Horde team, and it looks like a bug, one that will require careful consideration due to the potential performance regressions a fix may introduce.

The “-VerifyContent” approach seems like it would be much safer to add alongside our existing conform workflow.

The discussions are still ongoing internally, and I’ll let you know when we have a further update.

[Attachment Removed]