Seeing slow transfer rates for SendCas in UBA

Development Pipeline & Plugins
, ,
1

We’ve been using UBA machines set up in a DC in Germany for a while now and seen reasonable SendCas times to them even on the east coast of the US, but we’ve recently started setting up nodes in a DC in California and we’re seeing pretty weak transfer rates for it. From trying to sift through the code, if I understand right SendCas is the time it took to send the files to the agent that were missing from its cache, not the time to receive the outputs back, is that correct? Here are a couple examples, first is from NC to CA, second is from the same PC in NC to Germany, any advice appreciated.

North Carolina to California
  ProcessId: 63
  Start:     4.5s
  Duration:  3m52s

  ----------- Detours stats -----------
  Total                   200    198.3s
  WaitOnResponse          190     80.1s
  Host                            80.1s

  Attach                    1       8ms
  Detach                    1      <1ms
  Init                      1      <1ms
  CreateFile              164     73.8s
  CloseFile                 3      <1ms
  GetFullFileName           8     154ms
  DeleteFile                6      2.8s
  ListDirectory             7      3.5s
  Log                       1      <1ms
  SendFiles                 3    118.2s
  WriteFiles                3      <1ms
  FileTable                 1      <1ms
  DirTable                  1      <1ms

  Startup Time                      4ms
  Exit Time                      118.2s
  DetoursMem                      7.4mb
  IopsRead                          117
  IopsOther                         257
  PeakMem                         2.2gb
  CPU Time                        39.4s
  Wall Time                      110.1s

  ----------- Session stats -----------
  GetFileMsg               94     48.2s
  ListDirMsg                7      3.5s
  DeleteFileMsg             5      2.8s
  WaitGetFileMsg           65     18.7s
  CreateMmapFromFile       68      21ms
  WaitMmapFromFile         40     339ms

  ----------- Storage stats -----------
  CalculateCasKeys          3      13ms
  EnsureCas               159      13ms
  SendCas                   2    115.7s
     Bytes Raw/Comp    39.7mb     6.8mb
     Compress               6     114ms
  DecompressToMem          68       2ms

  ----------- Kernel stats ------------
  CreateFile              167      12ms
  CloseFile               135       9ms
  ReadFile                321      11ms
     Bytes                      217.6kb
  SetFileInfo               5      <1ms
  GetFileInfo             115      10ms
  CreateFileMapping         4      <1ms
  MapViewOfFile          1463       7ms
  CloseHandle            8467      10ms
  VirtualAlloc              5      <1ms
----------------------
North Carolina to Germany

ProcessId: 631
  Start:     1m4s
  Duration:  2m29s

  ----------- Detours stats -----------
  Total                    65    120.1s
  WaitOnResponse           55     91.0s
  Host                            91.0s

  CreateFile               36     90.5s
  SendFiles                 3     29.1s

  Startup Time                      5ms
  Exit Time                       29.3s
  DetoursMem                      7.1mb
  IopsRead                           66
  IopsOther                         179
  PeakMem                         1.8gb
  CPU Time                        55.5s
  Wall Time                      116.8s

  ----------- Session stats -----------
  GetFileMsg                5     10.0s
  DeleteFileMsg             5     480ms
  WaitGetFileMsg           26      3.2s
  CreateMmapFromFile        7      <1ms
  WaitMmapFromFile          1      1.7s

  ----------- Storage stats -----------
  CalculateCasKeys          3      16ms
  EnsureCas                31     75.6s
  SendCas                   2     27.4s
     Bytes Raw/Comp    20.4mb     3.6mb
     Compress               4     135ms
  DecompressToMem           7      <1ms

[Attachment Removed]

2

Steps to Reproduce
UBA related, can’t really provide a repro since it’d require setting up in the same DCs we’re using. :slight_smile:

[Attachment Removed]

3

“SendFiles” is the total amount of wall time it took to send the data back to the host (including decompressing/writing them to the right place). SendCas is sending the actual files (compressed) to the host (it populates in-memory casdb for those files)

The numbers look crazy high.. both fetching and sending. what kind of network connection is this? :slight_smile:

[Attachment Removed]

4

I don’t know if 5.6 has the dependency crawler stuff.. So I would recommend taking absolutely newest binaries from ue main if possible and test them.

[Attachment Removed]