Announcement

Collapse
No announcement yet.

Using BuildPatchServices with pak files

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Using BuildPatchServices with pak files

    Hello.

    How do I use the BuildPatchServices module to efficiently patch content inside pak files?

    I am trying to use BuildPatchServices to let mobile clients to download and update game content. The size of the content is a couple hundreds of MBs, so reducing the amount downloaded for each patch is a concern. But BuildPatchServices, which is what the new Mobile Patching Utilities in 4.14 uses underneath, doesn't seem to be aware of the files inside pak, and resorts to downloading the whole pak for each update. This is not better than just updating through the Google Play Store...
    Also, the Packaging process doesn't seem to allow not using pak files for mobile clients.

    I'll be so glad if I can be pointed to how to make updating more efficient.

    Thank you.


    Below is the actual log from an experimentation I did showing that the client downloads the whole pak when only one asset file is changed.

    This is after updating one asset file. But the client downloads 49MB which is much larger than the file that's changed.
    Note TotalDownloadedData.


    Code:
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: AppName: PatchTest2
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: AppInstalledVersion: Update10
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: AppPatchVersion: Update11
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: CloudDirectory: redacted
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumFilesInBuild: 1
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumFilesOutdated: 1
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumFilesToRemove: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumChunksRequired: 48
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ChunksQueuedForDownload: 48
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ChunksLocallyAvailable: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumChunksDownloaded: 48
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumChunksRecycled: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumChunksCacheBooted: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumDriveCacheChunkLoads: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumFailedDownloads: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumBadDownloads: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumRecycleFailures: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumDriveCacheLoadFailures: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: TotalDownloadedData: 49305602
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: AverageDownloadSpeed: 4.223 MB/sec
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: TheoreticalDownloadTime: 11.1 sec
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: VerifyTime: 120 ms
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: CleanUpTime: 1 ms
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: PrereqTime: 0 ms
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ProcessExecuteTime: 11.6 sec
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ProcessPausedTime: 0.0 sec
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ProcessSuccess: TRUE
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ErrorCode: OK
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: FailureReasonText: The operation was successful.
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: FailureType SUCCESS
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: NumInstallRetries: 0
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: FinalProgressValue: 1.000000
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: OverallRequestSuccessRate: 1.000000
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: ExcellentDownloadHealthTime: 11.393156
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: GoodDownloadHealthTime: 0.000000
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: OkDownloadHealthTime: 0.000000
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: PoorDownloadHealthTime: 0.000000
    [2016.10.12-13.28.34:525][801]BuildPatchServices: Build Stat: DisconnectedDownloadHealthTime: 0.000000
    Last edited by jaese; 11-29-2016, 12:45 PM.

    #2
    Hi,

    How you generate patch chunks? For example to generate patch Update11 you need to point BuildPatchServices to the same cloud directory as Update10, in this case it will use existing cloud directory to generate a diff. In general cloud directory should be always the same for all app patches, it acts as a pool of data from where BuildPatchServices can reconstruct any version of an app by a given manifest.

    Comment


      #3
      Thanks, ddvlost. Your tip is very effective!
      I made BuildPatchServices and BuildPatchTool.exe point to the same CloudDir for all updates and now I see that the client downloads only a fraction of the chunks that make up the pak. (from >300MB to 45MB)
      But the client still has to download 45MB. Do you suppose that I can reduce it further?

      Thanks


      Here are logs. The Content barely changed from update version 1.6.0 to 1.7.0. (a single data table)

      Initial content download:

      Code:
      [2016.11.30-06.24.32:710][  0]BuildPatchServices: Build Stat: AppName: redacted
      [2016.11.30-06.24.32:710][  0]BuildPatchServices: Build Stat: AppInstalledVersion: NONE
      [2016.11.30-06.24.32:710][  0]BuildPatchServices: Build Stat: AppPatchVersion: 1.6.0
      [2016.11.30-06.24.32:710][  0]BuildPatchServices: Build Stat: CloudDirectory: http://redacted/HTTPchunks/Android_ETC2/CloudDir
      [2016.11.30-06.24.32:711][  0]BuildPatchServices: Build Stat: NumFilesInBuild: 19
      [2016.11.30-06.24.32:711][  0]BuildPatchServices: Build Stat: NumFilesOutdated: 19
      [2016.11.30-06.24.32:711][  0]BuildPatchServices: Build Stat: NumFilesToRemove: 0
      [2016.11.30-06.24.32:711][  0]BuildPatchServices: Build Stat: NumChunksRequired: 863
      [2016.11.30-06.24.32:711][  0]BuildPatchServices: Build Stat: ChunksQueuedForDownload: 863
      [2016.11.30-06.24.32:711][  0]BuildPatchServices: Build Stat: ChunksLocallyAvailable: 0
      [2016.11.30-06.24.32:712][  0]BuildPatchServices: Build Stat: NumChunksDownloaded: 863
      [2016.11.30-06.24.32:712][  0]BuildPatchServices: Build Stat: NumChunksRecycled: 0
      [2016.11.30-06.24.32:712][  0]BuildPatchServices: Build Stat: NumChunksCacheBooted: 0
      [2016.11.30-06.24.32:712][  0]BuildPatchServices: Build Stat: NumDriveCacheChunkLoads: 0
      [2016.11.30-06.24.32:712][  0]BuildPatchServices: Build Stat: NumFailedDownloads: 0
      [2016.11.30-06.24.32:712][  0]BuildPatchServices: Build Stat: NumBadDownloads: 0
      [2016.11.30-06.24.32:713][  0]BuildPatchServices: Build Stat: NumRecycleFailures: 0
      [2016.11.30-06.24.32:713][  0]BuildPatchServices: Build Stat: NumDriveCacheLoadFailures: 0
      [2016.11.30-06.24.32:713][  0]BuildPatchServices: Build Stat: TotalDownloadedData: 379766101
      [2016.11.30-06.24.32:713][  0]BuildPatchServices: Build Stat: AverageDownloadSpeed: 6.269 MB/sec
      [2016.11.30-06.24.32:713][  0]BuildPatchServices: Build Stat: TheoreticalDownloadTime: 57.7 sec
      [2016.11.30-06.24.32:713][  0]BuildPatchServices: Build Stat: VerifyTime: 28.6 sec
      [2016.11.30-06.24.32:714][  0]BuildPatchServices: Build Stat: CleanUpTime: 2 ms
      [2016.11.30-06.24.32:714][  0]BuildPatchServices: Build Stat: PrereqTime: 0 ms
      [2016.11.30-06.24.32:714][  0]BuildPatchServices: Build Stat: ProcessExecuteTime: 1:52 min
      [2016.11.30-06.24.32:714][  0]BuildPatchServices: Build Stat: ProcessPausedTime: 0.0 sec
      [2016.11.30-06.24.32:719][  0]BuildPatchServices: Build Stat: ProcessSuccess: TRUE
      [2016.11.30-06.24.32:721][  0]BuildPatchServices: Build Stat: DisconnectedDownloadHealthTime: 0.000000
      [2016.11.30-06.24.32:719][  0]BuildPatchServices: Build Stat: ErrorCode: OK
      [2016.11.30-06.24.32:719][  0]BuildPatchServices: Build Stat: FailureReasonText: 
      [2016.11.30-06.24.32:719][  0]BuildPatchServices: Build Stat: FailureType SUCCESS
      [2016.11.30-06.24.32:720][  0]BuildPatchServices: Build Stat: NumInstallRetries: 0
      [2016.11.30-06.24.32:720][  0]BuildPatchServices: Build Stat: FinalProgressValue: 1.000000
      [2016.11.30-06.24.32:720][  0]BuildPatchServices: Build Stat: OverallRequestSuccessRate: 1.000000
      [2016.11.30-06.24.32:720][  0]BuildPatchServices: Build Stat: ExcellentDownloadHealthTime: 83.081268
      [2016.11.30-06.24.32:720][  0]BuildPatchServices: Build Stat: GoodDownloadHealthTime: 0.000000
      [2016.11.30-06.24.32:720][  0]BuildPatchServices: Build Stat: OkDownloadHealthTime: 0.000000
      [2016.11.30-06.24.32:721][  0]BuildPatchServices: Build Stat: PoorDownloadHealthTime: 0.000000
      Update to 1.7.0:

      Code:
      [2016.11.30-06.28.51:684][  0]BuildPatchServices: Verify stage complete 1
      [2016.11.30-06.28.51:684][  0]BuildPatchServices: Deleting staging area
      [2016.11.30-06.28.51:689][  0]BuildPatchServices: Reset MM
      [2016.11.30-06.28.51:689][  0]BuildPatchServices: Build Stat: AppName: redacted
      [2016.11.30-06.28.51:689][  0]BuildPatchServices: Build Stat: AppInstalledVersion: 1.6.0
      [2016.11.30-06.28.51:689][  0]BuildPatchServices: Build Stat: AppPatchVersion: 1.7.0
      [2016.11.30-06.28.51:690][  0]BuildPatchServices: Build Stat: CloudDirectory: http://redacted/HTTPchunks/Android_ETC2/CloudDir
      [2016.11.30-06.28.51:690][  0]BuildPatchServices: Build Stat: NumFilesInBuild: 19
      [2016.11.30-06.28.51:690][  0]BuildPatchServices: Build Stat: NumFilesOutdated: 1
      [2016.11.30-06.28.51:690][  0]BuildPatchServices: Build Stat: NumFilesToRemove: 0
      [2016.11.30-06.28.51:690][  0]BuildPatchServices: Build Stat: NumChunksRequired: 808
      [2016.11.30-06.28.51:690][  0]BuildPatchServices: Build Stat: ChunksQueuedForDownload: 129
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: ChunksLocallyAvailable: 679
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: NumChunksDownloaded: 129
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: NumChunksRecycled: 679
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: NumChunksCacheBooted: 0
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: NumDriveCacheChunkLoads: 0
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: NumFailedDownloads: 0
      [2016.11.30-06.28.51:691][  0]BuildPatchServices: Build Stat: NumBadDownloads: 0
      [2016.11.30-06.28.51:692][  0]BuildPatchServices: Build Stat: NumRecycleFailures: 0
      [2016.11.30-06.28.51:692][  0]BuildPatchServices: Build Stat: NumDriveCacheLoadFailures: 0
      [2016.11.30-06.28.51:692][  0]BuildPatchServices: Build Stat: TotalDownloadedData: 45024322
      [2016.11.30-06.28.51:692][  0]BuildPatchServices: Build Stat: AverageDownloadSpeed: 4.074 MB/sec
      [2016.11.30-06.28.51:692][  0]BuildPatchServices: Build Stat: TheoreticalDownloadTime: 10.5 sec
      [2016.11.30-06.28.51:692][  0]BuildPatchServices: Build Stat: VerifyTime: 28.6 sec
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: CleanUpTime: 2 ms
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: PrereqTime: 0 ms
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: ProcessExecuteTime: 1:24 min
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: ProcessPausedTime: 0.0 sec
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: ProcessSuccess: TRUE
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: ErrorCode: OK
      [2016.11.30-06.28.51:693][  0]BuildPatchServices: Build Stat: FailureReasonText: 
      [2016.11.30-06.28.51:694][  0]BuildPatchServices: Build Stat: FailureType SUCCESS
      [2016.11.30-06.28.51:694][  0]BuildPatchServices: Build Stat: NumInstallRetries: 0
      [2016.11.30-06.28.51:694][  0]BuildPatchServices: Build Stat: FinalProgressValue: 1.000000
      [2016.11.30-06.28.51:695][  0]BuildPatchServices: Build Stat: OverallRequestSuccessRate: 1.000000
      [2016.11.30-06.28.51:695][  0]BuildPatchServices: Build Stat: ExcellentDownloadHealthTime: 55.510036
      [2016.11.30-06.28.51:695][  0]BuildPatchServices: Build Stat: GoodDownloadHealthTime: 0.000000
      [2016.11.30-06.28.51:695][  0]BuildPatchServices: Build Stat: OkDownloadHealthTime: 0.000000
      [2016.11.30-06.28.51:695][  0]BuildPatchServices: Build Stat: PoorDownloadHealthTime: 0.000000
      [2016.11.30-06.28.51:695][  0]BuildPatchServices: Build Stat: DisconnectedDownloadHealthTime: 0.000000

      Comment


        #4
        This is probably result of non-deterministic cooking of some assets. If you package same content twice is your final pak files are same byte to byte?

        Comment


          #5
          No, pak files never stay the same between packaging.

          I compared Cooked asset files between packaging runs and found the results to be consistent: it's the .umap files that have different bytes for each cook.

          146 out of 150 .umap files are different.
          All .uasset files are identical.

          The .umap files make up about a hundred MBs, so I guess they are the reason for the less-than-optimal patches.
          Is there a cure for non-deterministic cooking?

          Comment


            #6
            Originally posted by jaese View Post
            No, pak files never stay the same between packaging.

            I compared Cooked asset files between packaging runs and found the results to be consistent: it's the .umap files that have different bytes for each cook.

            146 out of 150 .umap files are different.
            All .uasset files are identical.

            The .umap files make up about a hundred MBs, so I guess they are the reason for the less-than-optimal patches.
            Is there a cure for non-deterministic cooking?
            Could you make a simple project with one of those maps and share it? We are working on finding and fixing all those cases, so deterministic cooking improving with each release. Repro case will really help with it.

            Comment


              #7
              Hello,

              I'm also interested in the patching process for Android as I haven't been able to find much related to this in the documentation materials. I would be very interested to any pointers towards setting this up for the latest 4.14 version, especially using the newly announced Mobile Patch Utilities Blueprint library
              Ideally I would like the app to handle the patching process automatically as the user opts for more content to be loaded. I'm not clear on how the .manifest file should look like or how the UE4 patching should be set up with regards to the directory where the .pak file should be downloaded by the app.

              Any help would be very much appreciated.
              George
              George Cretu
              founder & executive CGArtefacts

              Marketplace content - Photo Studio Pack
              BP-Geographically-Correct-Sun-System

              Comment


                #8
                Originally posted by cgiteach View Post
                Hello,

                I'm also interested in the patching process for Android as I haven't been able to find much related to this in the documentation materials. I would be very interested to any pointers towards setting this up for the latest 4.14 version, especially using the newly announced Mobile Patch Utilities Blueprint library
                Ideally I would like the app to handle the patching process automatically as the user opts for more content to be loaded. I'm not clear on how the .manifest file should look like or how the UE4 patching should be set up with regards to the directory where the .pak file should be downloaded by the app.

                Any help would be very much appreciated.
                George
                If you follow these instructions: https://docs.unrealengine.com/latest...ard/index.html you should find all necessary files in generated content (also manifest).
                Realtime Database Firebase | Ultimate Mobile Kit Firebase | Extended Facebook | Universal Mobile Ads | Google Analytics

                Comment


                  #9
                  Thanks for the link, it definitely got me in the right direction, but I'm stuck at a different place now, and I've opened up a thread on this, maybe it could be of help to others trying to do the packing via BP only.
                  https://forums.unrealengine.com/show...atching-Mobile
                  Thanks again
                  George Cretu
                  founder & executive CGArtefacts

                  Marketplace content - Photo Studio Pack
                  BP-Geographically-Correct-Sun-System

                  Comment

                  Working...
                  X