We’re seeing shader symbol exports throw errors into the cooking log specifically when trying to open it for writing. Specifically:
FShaderSymbolExport::WriteFile
We can lower the verbosity of the logs to not cause our nightly builds to error out. But I wanted more information about the determinism of the PDBs. Investigating the issue I saw that the exporter checks the binary hash to make sure they’re equal, and if they’re not they take the smaller of the two hashes.
// when ensuring determinism, we further store the hash of the symbols we wrote out, and if the new hash is less than
// the previously written one we write it again, so we always choose the same copy of symbols to write during i.e. a cook
const FSymbolFileInfo* PrevExportedSymbolInfo = ExportedSymbolInfo.Find(Filename);
check(PrevExportedSymbolInfo);
check(NewSymbolHash != 0u);
if (NewSymbolHash >= PrevExportedSymbolInfo->Hash)
{
DuplicateSymbols.fetch_add(1, std::memory_order_relaxed);
return;
}
// Hash is lower; fall through to write phase to overwrite
I thought the PDB hash name was already calculated based on the shader binary. Have you seen PDBs with the same hash name return different binary hashes of the content?
Yes, shader symbols can definitely differ even if the symbol file name (generally some form of hash of the compiled bytecode) matches. You can reason about it if you consider what is contained in the symbol data - always some form of the original shader source - but compiler dead-stripping means that many different shader source inputs may result in identical output bytecode. They’re all equivalently useful for debugging in our experience since the dead code tends to be the main delta, though things like identical code with different variable names also could in theory be in play here. In addition to that, some platforms/compilers output purely non-deterministic symbols (out of our control, and not likely to ever be fixed) though this is not the norm thankfully.
Worth clarifying, it’s not that the shader source is non-deterministic - it’s that many different shaders end up generating the same compiled bytecode. What’s non deterministic is which such shader is the first to compile to any given unique bytecode and so win the “race” to generate a set of symbols.
I figured that the embedded source would be the main culprit. compilers not being deterministic isn’t unheard of either but I would have expected shaders to be simpler and more deterministic.