Cook fails with TextureReferenceIndex != INDEX_NONE (HLSLMaterialTranslator.cpp)

Development Pipeline & Plugins
, ,
1

Hi,

These cook errors happen roughly 1/5 times and prevent the cook from completing successfuly.

Assertion failed: TextureReferenceIndex != INDEX_NONE [File:…\Engine\Source\Runtime\Engine\Private\Materials\HLSLMaterialTranslator.cpp]

Similar to this editor issue: [Content removed]

Also already being tracked by Epic here: https://issues.unrealengine.com/issue/UE\-318809

The cooks fail on the exact same Material asset each time this occurs.

So far we have tried to:

* Resaving the asset, but this did not help.

* We observed that UMaterial::UpdateCachedExpressionData is indeed called prior to this happening.

* We’ve started tracking DDC Key files for this material during cooks with : r.ShaderCompiler.DebugDDCKeyAsset=MyProblematicMaterialAssetName

* On failed cooks we noticed that DDC Key file was missing 2 entries related to some code in FMaterialShaderMapId::RecordAndEmit:

\- ReferencedFunctions 

\- ReferencedParameterCollections

Since ReferencedFunctions and ReferencedParameterCollections are not written to the DDC keygen hash, the cook looks for an invalid hash key for that material’s DDC data.

Our current theory is that some of the Material’s dependencies (Expressions, textures parameters) are not yet fully loaded in when HLSLMaterialTranslator is called.

This prevents ReferencedFunctions and ReferencedParameterCollections from being populated prior to HLSLMaterialTranslator.

Is there a simple way to make sure UMaterial’s dependencies are loaded before it gets translated during the cook?

Would this make sense?

------------------------------------------ CODE------------------------------------------

UMaterial::UpdateCachedExpressionData()

//@note FH: temporary preemptive PostLoad until zenloader load ordering improvements

ConditionalPostLoad();

////////////////////////START/////////////////////////////////////////

for (UMaterialExpression* Expression : GetExpressions())

{

if (Expression)

{

Expression->ConditionalPostLoad();

}

}

////////////////////////END/////////////////////////////////////////

…

------------------------------------------ CODE------------------------------------------

Thanks,

David

[Attachment Removed]

2

Hi David,

Thank you for your feedback.

We are aware of this issue and been looking into a solution for a while. The problem indeed stems from indeterminism between the used MaterialFunction’s and the Material’s PostLoad. The function PostLoad may be in-flight and on the stack when the Material gets to PostLoad, tries to ConditionallyPostLoad its referenced functions but these are effectively no-ops because RF_NeedPostLoad gets set to false immediately by PostLoad and CPL earlies out.

Your hot fix is a good temporary solution and will probably eliminate your crash or greatly reduce it. I’d say ship it until we are able to make a broader more robust fix.

That said I’d suggest:

  1. Move /add the ConditionalPostLoad into FMaterialCachedExpressionData::UpdateForFunction instead, which covers the same case plus nested functions plus the MIC path with a single line in single choke point.
  2. A residual race remains (function postload suspended mid-stack) which no ConditionalPostLoad can fix, if you want to know whether their content ever hits it, add the DebugPostLoad checkf/ensure/log so it fails loud instead of cooking a bad DDC key.
// FMaterialCachedExpressionData::UpdateForFunction (MaterialCachedData.cpp)
  
  #include "UObject/UObjectThreadContext.h"
 
  void FMaterialCachedExpressionData::UpdateForFunction(const FMaterialCachedExpressionContext& Context, UMaterialFunctionInterface* Function, EMaterialParameterAssociation
  Association, int32 ParameterIndex)
  {
        if (!Function)
        {
                return;
        }
 
        // Zenloader guarantees imports are serialized before our PostLoad, but not postloaded.
        Function->ConditionalPostLoad();
 
  #if !(UE_BUILD_SHIPPING || UE_BUILD_TEST)
        // If its PostLoad is suspended on this callstack the CPL above was a no-op and the walk would harvest incomplete data.
        if (!ensureAlwaysMsgf(!FUObjectThreadContext::Get().DebugPostLoad.Contains(Function), TEXT("'%s' reached while its PostLoad is in progress (reentrant load pump)"), *Function->GetPathName()))
        {
                UE_LOG(LogMaterial, Error, TEXT("Cached expression data for this material will be incomplete: missing function/collection ids in the DDC key and missing referenced tex
  Function: '%s'"), *Function->GetPathName());
        }
  #endif
 
        ...

Hope this helps,

Massimo

[Attachment Removed]

3

Hi Massimo,

As per your suggestion, we implemented the fix in your code snippet.

Unfortunately the issue still occurs sporadically.

Thanks,

David

[Attachment Removed]

4

Hi David,

Sorry to hear that. Unfortunately the problem is deep in the Materials loading core logic and solving it will require quite a bit of research and changes. The core issue is that I identified three places where we have cyclic dependencies of PostLoads in Materials loading code which when they loop will cause a call to ConditionalPostLoad on a dependency that has already started it, and the new call will simply early out, pretending to have succesfully completed. Obviously this will cause Materials to have missing state.

I’ll be off for the coming three weeks for Summer Break. I can touch base again with you when I get back.

Thanks,

Massimo

[Attachment Removed]

5

Hi Massimo,

My fix still fails unfortunately. I will try the code snippet you suggest.

Is there an ETA for a fix to this longstanding issue? Please let us know.

Thanks,

David

[Attachment Removed]

6

Just to correctly clarify the situation we are seeing in on 5.5 (not 5.6).

Thanks,

David

[Attachment Removed]