When a horde gauntlet job runs a test that encounters an ensure, the crash handler’s DataRouterUrl is not hit. The provided repro steps can hopefully cover the finer details of this issue. Is there something in either Gauntlet code, or Horde etc. that prevents interaction with the CrashReportClient.exe? I feel like I’m missing something with regards to setup or crash handling. Let me know if this is something you guys encountered before!
Steps to Reproduce
- Create a gauntlet test, let’s use UE.BootTest as an example
- UE.BootTest is configured to start up one client
- Make sure your client ensures upon startup(?). This step is a bit different project to project. I just happened to have a build that did this so I’m using it as my test candidate.
- Make sure CrashReportClient’s DefaultEngine.ini is setup with a DataRouterUrl that is linked to your crash listener (we are using a python server that listens, unpacks and handles crashes)
- Run gauntlet via RunUAT.bat. Here’s an example of how we do it:
- RunUAT.bat RunUnreal -project=“<path to uproject>” -test=“UE.BootTest” -platform=Win64 -build=“<path to package build>” -nullrhi
- Wait for ensure
- Watch ensure go through the crash handler
- End gauntlet run
- Schedule a gauntlet job via Horde. Here’s a snapshot of our build graph that is invoked by the job template:
- <Node Name=“Gauntlet”>
- <Command Name=“RunUnreal” Arguments=“$(ProjectPath) $(BuildPath) $(Platform) -nullrhi -WriteTestResultsForHorde” />
- </Node>
- Schedule the new job
- Watch the Horde logs. See that ensures are being detected.
- Ensures do not go through the crash handler
Hey there Ralph,
Thanks for your question! I’ve corresponded internally with the subject matter experts of this area, and they provided some potential areas to look into.
- Gauntlet will create a clean user directory - so any local config is not going to be used unless it’s part of the project
- -BuildMachine could be relevant here, as it’s not reproducible locally
- https://github.com/EpicGames/UnrealEngine/blob/5.5/Engine/Source/Runtime/Core/Private/Windows/WindowsPlatformCrashContext.cpp#L910
- You could try passing -AllowCrashReportClientOnBuildMachine
- Through the buildgraph Gauntlet command you would need to add: -args=-AllowCrashReportClientOnBuildMachine to the Arguments property.
Kind regards,
Julian
[mention removed] your suggestion worked! They were being tagged as -BUILDMACHINE and so the crash context was returning early as you pointed out. Added that argument and now the crashes are reported. Thank you!!