Linux Server Crash Debug Issue: Missing Symbols and Minidumps
Problem Summary
Our Linux server builds are failing to generate usable crash diagnostics. We receive crash report archives, but they contain neither minidump files nor properly symbolicated callstacks, rendering the data virtually unusable for debugging purposes.
Example callstack:
0x000000000987268f NightfallServer-Linux-Test!?????????????()
0x0000000009868081 NightfallServer-Linux-Test!?????????????()
0x000000000985aad2 NightfallServer-Linux-Test!?????????????()
...
Troubleshooting Steps Completed
-
Fixed initial symbol generation failure (started with UE 5.4):
- Ever since our upgrade to UE 5.4, we’ve had to disable .sym generation because of a silent link-time crash generating .psyms, which cascaded down to a packaging error when .sym file could not be found.
- Identified that dump_syms.exe was failing due to DWARF v5 format in a 3rd party library
- Modified dump_syms.exe to handle DWARF v5, now successfully generating .psym files
-
Symbol verification:
- Confirmed .psym files appear well-formed
- Verified BreakpadSymbolEncoder can process the .psym files to generate .sym files
- Server is built with -g flag to include debug information
-
Runtime debugging:
- Traced through stackwalk code at runtime
- Observed that the code attempts to resolve symbols but fails consistently
- Symbol addresses from crash-time don’t match those in the .sym files
Current Status
Despite having properly packaged .sym files deployed alongside the binary, we cannot resolve any function names in the callstacks. The minidump files are also missing entirely.
Questions
- Is this a known issue with UE 5.4 on Linux?
- Are there additional steps required to enable proper minidump generation?
- What could cause the address mismatch between runtime crashes and our symbol files?
This issue began after upgrading to UE 5.4. Any guidance on resolving these Linux server crash debugging problems would be greatly appreciated.