Announcement

Collapse
No announcement yet.

4.19 & 4.20 Editor & Packaged Games Have Hangs/Hiccups Constantly

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Hey folks. We've been investigating this for a while with some help from Nvidia and have determined many of these leaks and hitches can be attributed to a bug in Aftermath that was exposed by a change in the way we render certain things. For the moment in 4.20 we will be disabling Aftermath by default ( you can still enable it manually if you need it ). The change is a simple one in Engine/Config/BaseEngine.ini to set r.gpucrashdebugging to False. I've confirmed that this fixes the leak (and so far in my soak the hitching) in Deathrey's project. If anyone is interested in testing this fix, make the change I mentioned then reboot your machine just to be sure, and run your projects again. You can confirm Aftermath is disabled in the log, or just by checking the state of 'r.gpucrashdebugging' via the console. (It should be 0). If anyone STILL sees a leak or hitching please report back as it's likely a separate but similar issue. Thanks to everyone in this thread for helping us track this down!
    Last edited by Waurelius; 07-11-2018, 11:40 AM. Reason: typo

    Comment


      Should the fix work in 4.19.2? I just tried and noticed no change :/ Below is the info from Dethrey's project with the render targets. (verified that r.gpucrashdebugging is 0, ran in standalone game)

      Time Since Boot: 20.23 Seconds

      Platform Memory Stats for Windows
      Process Physical Memory: 889.68 MB used, 926.53 MB peak
      Process Virtual Memory: 1177.03 MB used, 1226.25 MB peak
      Physical Memory: 5499.68 MB used, 10832.95 MB free, 16332.62 MB total
      Virtual Memory: 6117.48 MB used, 10832.95 MB free, 134217728.00 MB total


      Time Since Boot: 182.32 Seconds

      Platform Memory Stats for Windows
      Process Physical Memory: 1264.52 MB used, 1681.52 MB peak
      Process Virtual Memory: 1552.15 MB used, 1970.98 MB peak
      Physical Memory: 5823.97 MB used, 10508.65 MB free, 16332.62 MB total
      Virtual Memory: 6473.42 MB used, 10508.65 MB free, 134217728.00 MB total


      I also still had the original particle test project with the mesh emitters laying around and noticed the hitch within a minute or so.

      Comment


        spazfirem I took a look back and the .ini change I suggested is NOT sufficient for 4.19. Looks like 4.19 had a bug with the Aftermath initialization code that is kinda forcing it on. We straightened all that out in 4.20. Sorry about that, I hadn't realized that was going on. If you look in the log you will see "Aftermath enabled and primed" which indicates that it's on. Looks like for a binary release the only way to really kill it would be to delete the GFSDK_Aftermath_Lib.x64.dll from Binaries/ThirdParty/NVIDIA/NVaftermath/Win64/. If you can make code changes, then then forcing GDX11NVAfterMathEnabled to 0 in FD3D11DynamicRHIModule::StartupModule should resolve it.

        Comment


          Note that you'll need to do the .ini change AND the code change ( or just delete the dll ). As it's a bit of janky double-init logic in there for Aftermath.

          Comment


            Waurelius Thanks for the tip! Totally works.

            Time Since Boot: 20.98 Seconds

            Platform Memory Stats for Windows
            Process Physical Memory: 851.90 MB used, 878.93 MB peak
            Process Virtual Memory: 1136.92 MB used, 1207.85 MB peak
            Physical Memory: 5905.68 MB used, 10426.95 MB free, 16332.62 MB total
            Virtual Memory: 6074.63 MB used, 10426.95 MB free, 134217728.00 MB total


            Time Since Boot: 175.87 Seconds

            Platform Memory Stats for Windows
            Process Physical Memory: 852.74 MB used, 878.93 MB peak
            Process Virtual Memory: 1137.39 MB used, 1207.85 MB peak
            Physical Memory: 5876.44 MB used, 10456.18 MB free, 16332.62 MB total
            Virtual Memory: 6050.56 MB used, 10456.18 MB free, 134217728.00 MB total

            Seriously many thanks for tracking this down. Opened up the particle map and the stutter was gone as well. Haven't done a long test yet, but I'm confident.

            Just to confirm 4.19.2 with the .ini change and I just backed up the .dll and deleted it.

            Comment


              Waurelius Can confirm that extinguishing Aftermath deals with the issue (398.36, 4.19.2)
              Thanks and looking forward to it being fixed.

              Comment


                Im on 4.19.2, and tried the workaround with setting the Engine/Config/BaseEngine.ini r.gpucrashdebugging = False AND deleting the GFSDK_Aftermath_Lib.x64.dll from Binaries/ThirdParty/NVIDIA/NVaftermath/Win64/ i restarted my PC and now whenever i open my project, the engine crashes immediately, if i just start the engine to make a blank project, it crashes too. What may i have done wrong?

                Comment


                  On the other hand, i tried 4.20 and there it seem to run absolutely perfect, only problem i have, is that i rely on some thirdparty plugin that is obv. not updated to 4.20 yet, but it looks like this could solve it for me too.

                  Comment


                    Originally posted by StickySnoutStd View Post
                    Im on 4.19.2, and tried the workaround with setting the Engine/Config/BaseEngine.ini r.gpucrashdebugging = False AND deleting the GFSDK_Aftermath_Lib.x64.dll from Binaries/ThirdParty/NVIDIA/NVaftermath/Win64/ i restarted my PC and now whenever i open my project, the engine crashes immediately, if i just start the engine to make a blank project, it crashes too. What may i have done wrong?
                    for 4.19.2 it is necessary changes at the source code: "GDX11NVAfterMathEnabled to 0 in FD3D11DynamicRHIModule::StartupModule"
                    Nilson Lima
                    Technical Director @ Rigel Studios Ltda - twitter: @RigelStudios
                    Art is a state of Spirit

                    Join us at Discord: https://discord.gg/QJUb5Wk
                    UE4 Marketplace:
                    Cloudscape Seasons

                    Comment


                      Originally posted by Waurelius View Post
                      Note that you'll need to do the .ini change AND the code change ( or just delete the dll ). As it's a bit of janky double-init logic in there for Aftermath.
                      Shouldn't it be enough to only change

                      if (Target.Platform == UnrealTargetPlatform.Win64)

                      to

                      if (Target.Platform == UnrealTargetPlatform.Win64 && false)

                      In NVaftermath.build.cs?
                      Last edited by John Alcatraz; 07-11-2018, 04:23 PM.
                      Easy to use UMG Mini Map on the UE4 Marketplace.
                      Forum thread: https://forums.unrealengine.com/show...-Plug-and-Play

                      Comment


                        StickySnoutStd I would expect the LoadLibrary call to simply fail silently rather than crash but changing the code via my or John Alcatraz 's method should also work.

                        Comment


                          Originally posted by John Alcatraz View Post

                          Shouldn't it be enough to only change

                          if (Target.Platform == UnrealTargetPlatform.Win64)

                          to

                          if (Target.Platform == UnrealTargetPlatform.Win64 && false)

                          In NVaftermath.build.cs?
                          Yes this should work as well. You may have to force a rebuild if UBT doesn't pick up the modification for some reason.

                          Comment


                            There seems to be a bug. It is using:


                            Code:
                            void FD3D11DynamicRHIModule::StartupModule()
                            {
                            #if NV_AFTERMATH
                                // Note - can't check device type here, we'll check for that before actually initializing Aftermath
                            
                                    FString AftermathBinariesRoot = FPaths::EngineDir() / TEXT("Binaries/ThirdParty/NVIDIA/NVaftermath/Win64/");
                                    if (LoadLibraryW(*(AftermathBinariesRoot + "GFSDK_Aftermath_Lib.x64.dll")) == nullptr)
                                    {
                                        UE_LOG(LogD3D11RHI, Warning, TEXT("Failed to load GFSDK_Aftermath_Lib.x64.dll"));
                                        GDX11NVAfterMathEnabled = 0;
                                        return;
                                    }
                                    else
                                    {
                                        UE_LOG(LogD3D11RHI, Log, TEXT("Aftermath initialized"));
                                        GDX11NVAfterMathEnabled = 1;
                                    }
                            
                            #endif
                            }
                            But LoadLibraryW is documented as requiring backslashes:

                            https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx

                            Under "Remarks" it tells a complicated alternate search it will do if given a relative path that fails.

                            UE4 should maybe be using: IPlatformFile::ConvertToAbsolutePathForExternalAppForRead instead here?

                            Comment


                              Originally posted by Waurelius View Post

                              Yes this should work as well. You may have to force a rebuild if UBT doesn't pick up the modification for some reason.
                              Sadly i cant make engine changes because im relying on IKinema plugin. But as i read ur description its fixed in 4.20 and the most recent test i did with 4.20p5 with all Ikinema plugin functionaly disabled, it seemed to run smooth as it should.

                              Comment


                                Originally posted by Waurelius View Post
                                Hey folks. We've been investigating this for a while with some help from Nvidia and have determined many of these leaks and hitches can be attributed to a bug in Aftermath that was exposed by a change in the way we render certain things. For the moment in 4.20 we will be disabling Aftermath by default ( you can still enable it manually if you need it ). The change is a simple one in Engine/Config/BaseEngine.ini to set r.gpucrashdebugging to False. I've confirmed that this fixes the leak (and so far in my soak the hitching) in Deathrey's project. If anyone is interested in testing this fix, make the change I mentioned then reboot your machine just to be sure, and run your projects again. You can confirm Aftermath is disabled in the log, or just by checking the state of 'r.gpucrashdebugging' via the console. (It should be 0). If anyone STILL sees a leak or hitching please report back as it's likely a separate but similar issue. Thanks to everyone in this thread for helping us track this down!

                                Thank you VERY much, been waiting for this.
                                That's great news that it will be fixed in 4.20.

                                Comment

                                Working...
                                X