Announcement

Collapse
No announcement yet.

Dramatically reduce marketplace bandwidth and disk space usage with delta encoding

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Dramatically reduce marketplace bandwidth and disk space usage with delta encoding

    I already had Kite Demo 4.21 downloaded from the marketplace. To download the 4.23 version required a 24GB download.

    In contrast, to rsync between Kite Demo 4.21 and 4.23 only requires 4.3GB:

    Code:
    rsync -zrv --no-whole-file ABoyandHisKite423/ ABoyandHisKite421 --delete-after
    [...]
    sent 4,319,635,035 bytes  received 19,760,281 bytes  3,893,580.36 bytes/sec
    total size is 30,196,428,192  speedup is 6.96
    Compressed.ddp grew by around that same size between 4.21 and 4.23, (mayeb from adding a new platform?), and accounts for most of the data that was sent.

    This means in this case, downloading via marketplace is using around 6X more bandwidth than needed. rsync-style updating can have a lot of cpu usage, but with caching between common upgrades you wouldn't need to impact your servers more than once for each upgrade version-pair.

    The Vault Cache could also be stored delta encoded between engine version updates of the marketplace content and then people would be much less likely to clear it, saving much more bandwidth from re-downloads.

    This would both reduce your costs, and help your users, and the engine already has most of the infrastructure for it (implementing rolling hashes and everything) in:
    Code:
    Engine/Source/Runtime/Online/BuildPatchServices
    Last edited by muchcharles; 10-02-2019, 03:25 PM.
Working...
X