ImportMeshPaintTextureFromVertexColors only works with "Per Instance Vertex Colors"

Development Pipeline & Plugins
, ,
1

Hi there,

This is less of a question and more of an FYI + our code changes that fixes the issue.

Our artists wanted to use the “Import Vertex Colors” in the mesh paint mode and we found that it only worked if the instance had vertex color overrides enabled.

This is especially problematic for Nanite meshes since usually you can’t add actual vertex colors to the mesh.

As mentioned in the repro steps - the way to “fix” the vertex colors are as follows:

  • Disable Nanite on the source mesh
  • Paint a pixel of vertex color on the instance
  • Re-enable Nanite
  • “Import Vertex Colors” now works correctly.

I did a bit of digging and found the bit of code that was responsible for transferring the vertex colors and indeed it only considers per instance vertex colors.

The code in UMeshPaintModeSubsystem::ImportMeshPaintTextureFromVertexColors reads as follows:

	const bool bHasPerInstanceVertexColors = InstanceMeshLODInfo != nullptr && InstanceMeshLODInfo->OverrideVertexColors != nullptr;

	UE::Geometry::FStaticMeshLODResourcesToDynamicMesh::ConversionOptions ConversionOptions;
	ConversionOptions.bWantTangents = false;
	ConversionOptions.bWantMaterialIDs = false;

	UE::Geometry::FDynamicMesh3 DynamicMesh;
	UE::Geometry::FStaticMeshLODResourcesToDynamicMesh Converter;
	Converter.Convert(
		&StaticMesh->GetRenderData()->LODResources[LodIndex],
		ConversionOptions,
		DynamicMesh,
		bHasPerInstanceVertexColors,
		[InstanceMeshLODInfo](int32 Index)
		{
			return InstanceMeshLODInfo->OverrideVertexColors->VertexColor(Index);
		});

I’ve added a fallback check for the Nanite proxy to return the colors if possible:

	//#BFG_CHANGE Begin NANITE_VERTEX_COLORFIX
	const FColorVertexBuffer* BaseColorBuffer = nullptr;
	if (StaticMesh->GetRenderData()->LODResources.IsValidIndex(LodIndex))
	{
		const FStaticMeshLODResources& LODModel = StaticMesh->GetRenderData()->LODResources[LodIndex];
		BaseColorBuffer = &LODModel.VertexBuffers.ColorVertexBuffer;
	}


	UE::Geometry::FStaticMeshLODResourcesToDynamicMesh::ConversionOptions ConversionOptions;
	ConversionOptions.bWantTangents = false;
	ConversionOptions.bWantMaterialIDs = false;

	UE::Geometry::FDynamicMesh3 DynamicMesh;
	UE::Geometry::FStaticMeshLODResourcesToDynamicMesh Converter;
	Converter.Convert(
		&StaticMesh->GetRenderData()->LODResources[LodIndex],
		ConversionOptions,
		DynamicMesh,
		bHasPerInstanceVertexColors || (BaseColorBuffer != nullptr),
		[bHasPerInstanceVertexColors,InstanceMeshLODInfo, BaseColorBuffer](int32 Index)
		{
			if ( bHasPerInstanceVertexColors )
			{
				return InstanceMeshLODInfo->OverrideVertexColors->VertexColor(Index);
			}
			// fallback to the base vertex buffer 
			if ( BaseColorBuffer )
			{
				return BaseColorBuffer->VertexColor( Index );
			}
			return FColor::White;
		});
	//#BFG_CHANGE End

Kind Regards,

Johan

2

Steps to Reproduce
Repro of bug:

  • Import a mesh with vertex colors
  • Enable Nanite on the mesh
  • Place instance of the mesh in the scene
  • Switch to Mesh Paint mode
  • Switch to the Texture Color tab
  • Click “Import Vertex Colors”
  • Notice that it only returns pure white

Workaround:

  • Place a new instance (important as the previous step would cause the vertex colors to be wrong on the selected instance)
  • Disable Nanite on the mesh
  • Switch to Mesh Paint mode
  • Add some custom vertex color on the instance
  • Disable Nanite
  • Switch to Mesh Paint mode again
  • Switch to the Texture Color tab
  • Click “Import Vertex Colors”
  • Notice that it only returns pure white

I’ll try to create a repro project as a sample a bit later and upload it to the ticket.

3

Hello,

Apologies for the delay! I’ve verified the problem still exists in latest and the proposed fix works. I’ve created the following issue for tracking which should be visible publicly soon https://issues.unrealengine.com/issue/UE\-391452

Thank you for providing a fix!