Horde's BatchOUtputWriter is very slow for very large files

Development Pipeline & Plugins
, ,
1

This is a follow-up to [Content removed]

5.7 appears to have had some changes to BatchOutputWriter, but we are still seeing this slow behaviour. I’ve updated the PR with changes we have been running since 5.7.0 release that fixes this for us. This most recent version is narrower, and hopefully easier to review and integrate: https://github.com/EpicGames/UnrealEngine/pull/13349

2

Steps to Reproduce

3

Hello, thank you for the PR. This is now in our backlog to review. Likely will be some minor edits to placed the change behind a feature flag allowing us and other licensees to opt-in to the change.

Matthew

4

I am making a note to try this out in a test environment myself, as although we have much faster throughput in general, rather than extremely slow continuous, we also have extremely long lagtimes as the disk becomes completely saturated, so i’m guessing that writeback cache is making it look faster than what it’s really doing, until it’s full.

```

Written 352 files (4812.3mb, 1157.4mb/s)

Written 688 files (13950.8mb, 1646.8mb/s)

Written 1070 files (22745.4mb, 1610.1mb/s)

Written 2040 files (30926.6mb, 1631.4mb/s)

Written 2573 files (38903.2mb, 1686.4mb/s)

Written 2946 files (45555.9mb, 1084.8mb/s)

Written 3187 files (50189.7mb, 665.7mb/s)

Written 3304 files (53108.8mb, 557.1mb/s)

Written 3358 files (54168.4mb, 248.8mb/s)

Written 3451 files (56170.0mb, 464.4mb/s)

Written 3451 files (56170.0mb, 0.0mb/s)

Written 3451 files (56170.0mb, 0.0mb/s)

Written 3542 files (57687.3mb, 401.2mb/s)

Written 3659 files (59582.2mb, 524.1mb/s)

Written 3752 files (61345.9mb, 228.7mb/s)

Written 3752 files (61345.9mb, 0.0mb/s)

Written 3752 files (61345.9mb, 0.0mb/s)

```

not sure exactly when i’ll get to trying that out, though, as i don’t really have a system setup to build custom horde instances yet, but i’ll try to remember to pop back on this thread and give feedback

5

Hi closing this one, we have the PR and will be looking to add to 5.8

Thanks

Matthew

6

The speed discrepancy should also be observable when using UGS/Horde CMD to download with a `.uartifact` file. That’s how I was “integration testing” before deploying to the fleet.