Getting lots of POST time outs with Zen Server Shared DDC

Development Pipeline & Plugins
, ,
1

We recently set up a Zen Served based Shared DDC behind our VPN and in generally it has been fine, but especially when performing a fresh cook of a lot of data, like standing up a new branch, we get an insane number of POST time outs. From what I can see nothing on our Zen machine is even breathing heavy. Memory, CPU, storage, network are all great. On the client machine that is logging the errors though they are often performing a lot of local cooking, which means heavy CPU pressure and the like. We cook using four workers, not sure if that is a factor or not. The errors we see are basically these two (and so so many of them)

01:43:34 LogDerivedDataCache: Display: ZenShared: Error response received from GetCacheRecords RPC: from POST http://100.117.185.74:8558/z$/$rpc: Recv failure: Connection was reset.“Payload (application/x-ue-cbpkg): 6555726 bytes”

01:43:34 LogDerivedDataCache: Display: ZenShared: Error response received from GetCacheRecords RPC: from POST http://100.117.185.74:8558/z$/$rpc: Operation timed out after 301062 milliseconds with 0 bytes received.“Payload (*/*): 0 bytes”

I’m additionally uncertain where that timeout is set to modify it, but I imagine waiting five minutes is probably causing a fair share of issues of its own.

Any help appreciated!

[Attachment Removed]

2

Steps to Reproduce

  1. Set up a new stream
  2. Set up Zen Server shared DDC on another host
  3. Cook against the Zen Server using CookProcessCount=4
  4. Note that it will spam POST failures like the two below:

01:43:34 LogDerivedDataCache: Display: ZenShared: Error response received from GetCacheRecords RPC: from POST http://100.117.185.74:8558/z$/$rpc: Recv failure: Connection was reset.“Payload (application/x-ue-cbpkg): 6555726 bytes”

01:43:34 LogDerivedDataCache: Display: ZenShared: Error response received from GetCacheRecords RPC: from POST http://100.117.185.74:8558/z$/$rpc: Operation timed out after 301062 milliseconds with 0 bytes received.“Payload (*/*): 0 bytes”

[Attachment Removed]

3

I have not encountered something like this before. You said this was over VPN - is there perhaps something (whether it is VPN or a proxy) that is getting in the middle of this operation? Can you see signs in the zenserver log (not the Unreal log, but the one on the zenserver host itself) that show signs that it was operating on these requests but aborted/failed?

[Attachment Removed]