Question about -forceAllowShaderCompilerJobCache and the impact of IsShaderJobCacheDDCEnabled()

Development Asset Creation
, ,
1

Hi,

We have a number of custom commandlets which run with -AllowCommandletRendering, we’re noticing that these commandlets seem to be always building shaders even when those materials should be cached (ie from viewing the content in the editor on the same machine). After some investigation, we’ve narrowed it down the fact that the DDC ids end up being different we think due to this code (although it’s possible there is also another codepath which results in different IDs)

static bool IsRunningCookCommandletAndNotODSCServer()
{
	static const bool bIsCookCommandlet = IsRunningCookCommandlet();
	static const bool bIsODSCServer = FParse::Param(FCommandLine::Get(), TEXT("odsc"));
	return bIsCookCommandlet && !bIsODSCServer;
}

bool IsShaderJobCacheDDCEnabled()
{
#if WITH_EDITOR
	static const bool bForceAllowShaderCompilerJobCache = FParse::Param(FCommandLine::Get(), TEXT("forceAllowShaderCompilerJobCache"));
	static const bool bEnablePerShaderDDCCook = IsRunningCookCommandlet() && CVarShaderCompilerPerShaderDDCCook.GetValueOnAnyThread();
	static const bool bIsNonCookCommandlet = IsRunningCommandlet() && !IsRunningCookCommandlet();
#else
	const bool bForceAllowShaderCompilerJobCache = false;
	const bool bEnablePerShaderDDCCook = false;
	const bool bIsNonCookCommandlet = false;
#endif

	// Enable remote per-shader DDC for editor, game, cooks (if cvar is set), and for other commandlets only if the force flag is set on the cmdline.
	if ((GIsEditor || IsRunningGame() || bEnablePerShaderDDCCook) && (!bIsNonCookCommandlet || bForceAllowShaderCompilerJobCache))
	{
		// job cache itself must be enabled first
		return CVarJobCacheDDC.GetValueOnAnyThread();
	}

	return false;
}

bool IsMaterialMapDDCEnabled()
{
	// If we are loading individual shaders from the shader job cache for ODSC, don't attempt to load full material maps. 
	// Otherwise always load/cache material maps in cooks.
	return (IsShaderJobCacheDDCEnabled() == false) || IsRunningCookCommandletAndNotODSCServer();
}

This is also problematic since we use the UDerivedDataCacheCommandlet with -fill to try and precompile materials and other assets on developers machines overnight, however due to this code it ends up caching a different variant of materials than what the editor will eventually use.

We were wondering what the goal of this code was and why it has special handling for the cook commandlet. Is there a danger to passing -forceAllowShaderCompilerJobCache to our commandlets? This repeated compilation is especially problematic for UWorldPartitionBuilders run locally, like HLOD builds for example, since it means the user essentially needs to build all materials for the entire world a second time which really slows down iteration.

Thanks,

Lucas

[Attachment Removed]

2

There’s no danger here, effectively this _should_ mean that for such commandlets only full material shader maps will be cached; I’d suggest confirming that’s the case and there’s not some real DDC key mismatch problem occurring here.

This code was added as a safety measure as we haven’t tested performance of per-shader caching in all possible scenarios - if it works there’s no downside to forcing it on (we do this in a few places, the DDC fill commandlet is one such place).

[Attachment Removed]

3

Thanks for the reply! Ok good to know we can just enable it then.

I did confirm they are cached, so nothing is broken, but it means that we basically need to compile everything twice, once for the cook commandlet and once for the other commandlet so it uses more build resources.

It feels like something that maybe should be on by default for FillDDC though just to avoid confusion.

[Attachment Removed]