Painting Vertex Colours From Code

AlanEdwardes · July 1, 2014, 12:41am

Hi,

This is a repost from the C++ Gameplay Programming forum, as I wasn’t getting any answers there and realised that might not be the best place for this question! The original thread is here: https://forums.unrealengine.com/showthread.php?13885-Painting-Vertex-Colours-From-Code

I’m trying to write some code that traverses vertices within a certain distance from a given vert, and changes their vertex colour.

Here’s a quick illustration, the green vert in the centre “spreads” to the vertices around it, making them green:

The code that finds the verts within the distance is fairly simple, but I’m having trouble actually changing the colour of vertices that I have found.

The UStaticMeshComponent->StaticMesh member has the GetVertexColorData and SetVertexColorData methods, which work, but aren’t a good solution (it is to be used in-game, so won’t actually affect the mesh, just instances of the mesh).

The only example of vertex painting I can find in the engine source is the mesh paint tool in the editor: MeshPaintEdMode.cpp

I have been trying to pick that apart, but I’m a little stuck (I’m not sure which portion of the code applies to what I am trying to do). Here’s what I have:


ColorSpreadComponent->SetLODDataCount(0 + 1, ColorSpreadComponent->LODData.Num());

FStaticMeshComponentLODInfo* InstanceMeshLODInfo = &ColorSpreadComponent->LODData[0];

InstanceMeshLODInfo->OverrideVertexColors = new FColorVertexBuffer;

FStaticMeshLODResources& LODModel = ColorSpreadComponent->StaticMesh->RenderData->LODResources[0];

// Init all colours to green as a test
if ((int32)LODModel.ColorVertexBuffer.GetNumVertices() >= LODModel.GetNumVertices())
{
	InstanceMeshLODInfo->OverrideVertexColors->InitFromColorArray(&LODModel.ColorVertexBuffer.VertexColor(0), LODModel.GetNumVertices());
}
else
{
	InstanceMeshLODInfo->OverrideVertexColors->InitFromSingleColor(FColor::Green, LODModel.GetNumVertices());
}

// Loop all of the verts and set their colour to green (this would be the "spread" code, and it would conditionally pick out verts)
for (uint32 i = 0; i < InstanceMeshLODInfo->OverrideVertexColors->GetNumVertices(); i++)
{
	InstanceMeshLODInfo->OverrideVertexColors->VertexColor(i) = FColor::Green;
}

It happily runs, but doesn’t do anything

ColorSpreadComponent is a UStaticMeshComponent, and for now I’m just trying to colour all vertices green as a place to start. The mesh in question already has vertex colours set up.

I’m also running this inside an AActor derived class, inside the Tick method, which is probably the wrong place to be doing this stuff.

Can anyone point me in the right direction?

Cheers.

anonymous_user_f2b8b290 · July 1, 2014, 1:00am

It’s possible to do what you want, although it will be a bit tricky and may end up performing poorly.

OverrideVertexColors is your ticket, that’s how you can modify vertex colors on just a UStaticMeshComponent at runtime and not modify the UStaticMesh.

Call BeginInitResource(OverrideVertexColors) after modifying to send the new values to the GPU. It may get done in UStaticMeshComponent::InitResources() anyway, I can’t tell. Setup a material which outputs VertexColor to Emissive for debugging.

That’s the only thing I can see that’s missing.

AlanEdwardes · July 1, 2014, 10:38am

Thanks for the help .

I tried adding a call to [FONT=Lucida Console]BeginInitResource(InstanceMeshLODInfo->OverrideVertexColors); after modifying the vertex colours, but there must be something else too as it still doesn’t change.

To make this easier, I’ll post the full code below (note to anyone browsing this thread, do not use this for anything - it is horrible test code that eats babies :)):

VertexColorSpreadMesh.h


#pragma once

#include "GameFramework/Actor.h"
#include "VertexColorSpreadMesh.generated.h"

UCLASS()
class AVertexColorSpreadMesh : public AActor
{
	GENERATED_UCLASS_BODY()
	
	UPROPERTY(BlueprintReadOnly, VisibleAnywhere, Category = "MeshActor")
	TSubobjectPtr<UStaticMeshComponent> ColorSpreadComponent;

	virtual void Tick(float DeltaTime) override;
};

VertexColorSpreadMesh.cpp


#include "VertexColorSpreadMesh.h"

AVertexColorSpreadMesh::AVertexColorSpreadMesh(const class FPostConstructInitializeProperties& PCIP)
	: Super(PCIP)
{
	PrimaryActorTick.bStartWithTickEnabled = true;
	PrimaryActorTick.bCanEverTick = true;

	ColorSpreadComponent = PCIP.CreateDefaultSubobject<UStaticMeshComponent>(this, TEXT("ColorSpreadMesh"));
	RootComponent = ColorSpreadComponent;
}

void AVertexColorSpreadMesh::Tick(float DeltaTime)
{
	Super::Tick(DeltaTime);

	GEngine->AddOnScreenDebugMessage(-1, 5.0f, FColor::Yellow, FString::Printf(TEXT("Verts %d: "), ColorSpreadComponent->StaticMesh->GetNumVertices(0)));

	ColorSpreadComponent->SetLODDataCount(1, ColorSpreadComponent->LODData.Num());

	FStaticMeshComponentLODInfo* InstanceMeshLODInfo = &ColorSpreadComponent->LODData[0];

	InstanceMeshLODInfo->PaintedVertices.Empty();

	InstanceMeshLODInfo->OverrideVertexColors = new FColorVertexBuffer;

	FStaticMeshLODResources& LODModel = ColorSpreadComponent->StaticMesh->RenderData->LODResources[0];

	if ((int32)LODModel.ColorVertexBuffer.GetNumVertices() >= LODModel.GetNumVertices())
	{
		InstanceMeshLODInfo->OverrideVertexColors->InitFromColorArray(&LODModel.ColorVertexBuffer.VertexColor(0), LODModel.GetNumVertices());
	}
	else
	{
		InstanceMeshLODInfo->OverrideVertexColors->InitFromSingleColor(FColor::Green, LODModel.GetNumVertices());
	}

	for (uint32 i = 0; i < InstanceMeshLODInfo->OverrideVertexColors->GetNumVertices(); i++)
	{
		InstanceMeshLODInfo->OverrideVertexColors->VertexColor(i) = FColor::Green;
	}

	BeginInitResource(InstanceMeshLODInfo->OverrideVertexColors);
}

Screenshot of the shader and mesh I’m using to test this:

Edit: And the FBX I’m using here: http://s.alanedwardes.com/SM_DevVertexColorWall.fbx

Cheers.

anonymous_user_f2b8b290 · July 1, 2014, 11:45pm

Oh God, it’s in Tick =)

Try MarkRenderStateDirty() after modifying OverrideVertexColors. I didn’t realize this was in tick, once the game has started. That will notify the rendering thread that the component has changed.

Does the component you’re creating show up at all? Is that what’s in the screenshot, or is that a different component using the same UStaticMesh?

If it still doesn’t work, some debugging is going to be needed. You want to see whether OverrideVertexColors is getting applied properly in here

FStaticMeshSceneProxy::FLODInfo::FLODInfo(

if( ComponentLODInfo.OverrideVertexColors )
{
FStaticMeshLODResources& LODRenderData = RenderData->LODResources[LODIndex];

		// the instance should point to the loaded data to avoid copy and memory waste
		OverrideColorVertexBuffer = ComponentLODInfo.OverrideVertexColors;

		// Setup our vertex factory that points to our overridden color vertex stream.  We'll use this
		// vertex factory when rendering the static mesh instead of it's stock factory
		OverrideColorVertexFactory.Reset( new FLocalVertexFactory() );
		LODRenderData.InitVertexFactory( *OverrideColorVertexFactory.GetOwnedPointer(), InComponent-&gt;StaticMesh, OverrideColorVertexBuffer );

		// @todo MeshPaint: Make sure this is the best place to do this; also make sure cleanup is called!
		BeginInitResource( OverrideColorVertexFactory.GetOwnedPointer() );
	}

AlanEdwardes · July 3, 2014, 1:44pm

That worked perfectly! Video with a bit of a delay added to the colour change: http://s.edward.es/2014-07-03-5o0G2X.mp4

Here is what the full code looks like now (the code still eats babies and is only suitable for testing):

VertexColorSpreadMesh.h


#pragma once

#include "GameFramework/Actor.h"
#include "VertexColorSpreadMesh.generated.h"

UCLASS()
class AVertexColorSpreadMesh : public AActor
{
	GENERATED_UCLASS_BODY()
	
	UPROPERTY(BlueprintReadOnly, VisibleAnywhere, Category = "MeshActor")
	TSubobjectPtr<UStaticMeshComponent> ColorSpreadComponent;

	virtual void Tick(float DeltaTime) override;
};

VertexColorSpreadMesh.cpp


#include "VertexColorSpreadMesh.h"

AVertexColorSpreadMesh::AVertexColorSpreadMesh(const class FPostConstructInitializeProperties& PCIP)
	: Super(PCIP)
{
	PrimaryActorTick.bStartWithTickEnabled = true;
	PrimaryActorTick.bCanEverTick = true;

	ColorSpreadComponent = PCIP.CreateDefaultSubobject<UStaticMeshComponent>(this, TEXT("ColorSpreadMesh"));
	RootComponent = ColorSpreadComponent;
}

void AVertexColorSpreadMesh::Tick(float DeltaTime)
{
	Super::Tick(DeltaTime);

	GEngine->AddOnScreenDebugMessage(-1, 5.0f, FColor::Yellow, FString::Printf(TEXT("Verts %d: "), ColorSpreadComponent->StaticMesh->GetNumVertices(0)));

	ColorSpreadComponent->SetLODDataCount(1, ColorSpreadComponent->LODData.Num());

	FStaticMeshComponentLODInfo* InstanceMeshLODInfo = &ColorSpreadComponent->LODData[0];

	InstanceMeshLODInfo->PaintedVertices.Empty();

	InstanceMeshLODInfo->OverrideVertexColors = new FColorVertexBuffer;

	FStaticMeshLODResources& LODModel = ColorSpreadComponent->StaticMesh->RenderData->LODResources[0];

	if ((int32)LODModel.ColorVertexBuffer.GetNumVertices() >= LODModel.GetNumVertices())
	{
		InstanceMeshLODInfo->OverrideVertexColors->InitFromColorArray(&LODModel.ColorVertexBuffer.VertexColor(0), LODModel.GetNumVertices());
	}
	else
	{
		InstanceMeshLODInfo->OverrideVertexColors->InitFromSingleColor(FColor::Green, LODModel.GetNumVertices());
	}

	**BeginInitResource(InstanceMeshLODInfo->OverrideVertexColors);**
	**ColorSpreadComponent->MarkRenderStateDirty();**

	for (uint32 i = 0; i < InstanceMeshLODInfo->OverrideVertexColors->GetNumVertices(); i++)
	{
		InstanceMeshLODInfo->OverrideVertexColors->VertexColor(i) = FColor::Green;
	}
}

Now to make the colour spread, but that’s the fun bit Thanks a lot for the help.

AlanEdwardes · July 4, 2014, 2:40pm

And here’s the finished product: http://s.edward.es/2014-07-04-0N6a6H.mp4

AlanEdwardes · July 25, 2014, 10:59am

Thought I’d post a quick update about this - here’s a YouTube vid showing the vertex colours being used for opacity masking, and with particles being spawned where the intensity spreads: https://www.youtube.com/watch?v=jRAf9jx572w

I’d be open to tidying up and GitHubbing the code if anyone would find this useful for their stuff.

Omberone · October 17, 2014, 10:46am

I get a linker-error with BeginInitResource. Do I need to include any modules for it to have a definition?

EDIT: Found the modules; “RenderCore” and “ShaderCore”. New problem though; this function runs extremely slowly. Are there any optimizations in place?

Omberone · October 17, 2014, 3:07pm

Also, OverrideVertexColors = new FColorVertexBuffer; leaks. A lot.

JBaldwin · October 19, 2014, 4:03am

Fantastic! This looks like a lot of fun
Thanks for posting your progress in code for the community to see.

-Jeremy-

AlanEdwardes · October 8, 2015, 11:01pm

Sorry to ressurect a dead thread, but I have updated this code to the latest engine version and thrown it on GitHub as a few people expressed interest: GitHub - alanedwardes/UE4VertexColorSpread: Vertex colour spread code for Unreal Engine 4.

It is probably terrible code, however if anyone wants to submit pull requests to improve it I’ll gladly accept them.

Cheers

roushenzo · March 23, 2016, 5:50am

Thanks man !!! I was just looking for this solution ! I do not know much C ++, but it is the only way that you suggested !

plojnitza · June 28, 2016, 2:04pm

Does it work when shipping ? For a shippable build with cooked data we get crashes

LuisAlaguna · April 18, 2019, 1:10pm

Is it possible to use this to paint landscapes?