I had to edit the original code to be able to choose the UV index
This is generally right, but in practice, for accuracy (the accuracy of the float type in UE4 and if your game world is very large), the pickup should be done in the actor local coordinate system, rather than transforming back to the world coordinate system.
1 Like
Here is an alternative way to do this, in case anyone still wants to get UVs from skeletal meshes:
1 Like
Hi, does this actually work for you? I recreated your graph, but the mesh gets painted all over instead of the location I actually want. Could you share your code source?
Edit: Okay got it working. But as @BarrierAbyss said, the performance is very bad, especially if you have a high poly character.
I think even the above solution won’t work. I didn’t have luck with it. I think it might work for skeletal meshes that have 100% weights to a single bone for every vert, which works well on the default mannequin, but not a real skinned mesh. But I could be wrong, curious if you guys got it working reliably, for more complex meshes.
Example, I would imagine if every vert on the mannequins lower arm has 100% weight applied to the lowerarm joint, then any transform to the lowerarm bone should help you convert the vert from world space hit, backwards to your reference space pose just by unwinding the bone transform as you have done.
But if you don’t have that case, you would need to cumulatively add and apply all the weights for each bone that vert is associated with in order to reverse the animated pose. I’m sure its possible, but I’m lazy. Most skinned meshes have verts with weight from numerous bones.
Here is a method that for sure works, but as above, its very expensive:
Note that you need to have Per Poly Collision enabled on your skeletal mesh to do this, otherwise the Hit.FaceIndex will return -1;
USkeletalMeshComponent * Mesh = Cast<USkeletalMeshComponent>(Hit.Component);
TArray<FFinalSkinVertex> SkinnedVerts;
Mesh->GetCPUSkinnedVertices(SkinnedVerts, 0);
FSkeletalMeshRenderData* RenderData = Mesh->GetSkeletalMeshRenderData();
FRawStaticIndexBuffer16or32Interface * Indices = RenderData->LODRenderData[0].MultiSizeIndexContainer.GetIndexBuffer();
if(Hit.FaceIndex * 3 + 2 > Indices->Num()) return false;
const int32 Index0 = Indices->Get(Hit.FaceIndex * 3 + 0);
const int32 Index1 = Indices->Get(Hit.FaceIndex * 3 + 1);
const int32 Index2 = Indices->Get(Hit.FaceIndex * 3 + 2);
const FVector Pos0 = SkinnedVerts[Index0].Position;
const FVector Pos1 = SkinnedVerts[Index1].Position;
const FVector Pos2 = SkinnedVerts[Index2].Position;
const FVector2D UV0 = FVector2D(SkinnedVerts[Index0].U, SkinnedVerts[Index0].V);
const FVector2D UV1 = FVector2D(SkinnedVerts[Index1].U, SkinnedVerts[Index1].V);
const FVector2D UV2 = FVector2D(SkinnedVerts[Index2].U, SkinnedVerts[Index2].V);
const FVector LocalHitPos = Mesh->GetComponentToWorld().InverseTransformPosition(Hit.Location);
const FVector BaryCoords = FMath::ComputeBaryCentric2D(LocalHitPos, Pos0, Pos1, Pos2);
const FVector2D UV = (BaryCoords.X * UV0) + (BaryCoords.Y * UV1) + (BaryCoords.Z * UV2);
return true;
You are probably better off just using the unwrapping that Ryan Brucks showcased and MargraveUnreal suggested. I have tried that method also, and once you dilate the UV’s to avoid seams it works nicely, and its largely done using the GPU so it avoids looping through a bunch of verts, or transfering data from GPU->CPU.
Okay, found an even better solution. Not sure why I couldn’t find this function before but inside SkinnedMeshComponent there is a function called GetSkinnedVertexPosition() which gets exactly what we need to compute the UVs.
{
USkeletalMeshComponent* Mesh = Cast<USkeletalMeshComponent>(Hit.Component);
FSkeletalMeshRenderData* RenderData = Mesh->GetSkeletalMeshRenderData();
FSkeletalMeshLODRenderData& LODData = RenderData->LODRenderData[0];
FRawStaticIndexBuffer16or32Interface* Indices = RenderData->LODRenderData[0].MultiSizeIndexContainer.GetIndexBuffer();
if(Hit.FaceIndex * 3 + 2 > Indices->Num() || Hit.FaceIndex * 3 < 0) return false;
const int32 Index0 = Indices->Get(Hit.FaceIndex * 3 + 0);
const int32 Index1 = Indices->Get(Hit.FaceIndex * 3 + 1);
const int32 Index2 = Indices->Get(Hit.FaceIndex * 3 + 2);
const FVector Pos0 = Mesh->GetSkinnedVertexPosition(Mesh, Index0, LODData, RenderData->LODRenderData[0].SkinWeightVertexBuffer);
const FVector Pos1 = Mesh->GetSkinnedVertexPosition(Mesh, Index1, LODData, RenderData->LODRenderData[0].SkinWeightVertexBuffer);
const FVector Pos2 = Mesh->GetSkinnedVertexPosition(Mesh, Index2, LODData, RenderData->LODRenderData[0].SkinWeightVertexBuffer);
const FVector2D UV0 = Mesh->GetVertexUV(Index0, 0);
const FVector2D UV1 = Mesh->GetVertexUV(Index1, 0);
const FVector2D UV2 = Mesh->GetVertexUV(Index2, 0);
const FVector LocalHitPos = Mesh->GetComponentToWorld().InverseTransformPosition(Hit.Location);
const FVector BaryCoords = FMath::ComputeBaryCentric2D(LocalHitPos, Pos0, Pos1, Pos2);
UV = (BaryCoords.X * UV0) + (BaryCoords.Y * UV1) + (BaryCoords.Z * UV2);
return true;
}
This is much more performant than using my previous CPU skinning example. Again, you will need PerPolyCollision enabled on the SK mesh, but you will get more accurate calculations like this compared to testing against collision capsules which is important if you are using these UVs for things like decals/mesh paint etc.
1 Like
This is really awesome! Performance is great, too. One thing that is bothering me is, that you need uniform UVs to have the decals being the same size regardless of where we hit our character. For example the face of my character uses a lot more space on my UV map than the arms for example. This leads to tiny decals on the characters face and giant ones on his arms. Is there any way to get the UV stretch amount, so we can correct the decal scale?
Edit: Okay, this somewhat gives satisfying results to account for UV stretching. I calculate the area of the UV triangle and vertex triangle and calculate the stretch as the ratio of their square roots. Not sure if this is the correct way to do it. I still need to ease in the result to get a good ratio between stretched and non stretched parts of my UV map.
/*Calculate the UV stretch*/
//Calculate area of 3D triangle
FVector3f side1_3d = Pos1 - Pos0;
FVector3f side2_3d = Pos2 - Pos0;
FVector3f crossProduct_3d = FVector3f::CrossProduct(side1_3d, side2_3d);
float area3D = 0.5f * crossProduct_3d.Size();
//Calculate area of 2D triangle
FVector2D side1_2d = UV1 - UV0;
FVector2D side2_2d = UV2 - UV0;
float crossProduct_2d = FVector2D::CrossProduct(side1_2d, side2_2d);
float area2D = FMath::Abs(crossProduct_2d) * 0.5f;
// Calculate stretch
stretch = sqrt(area2D) / sqrt(area3D);
Hey hGosling! Cool find with the stretch! Did you manage to use the result to fix the decal scaling? If so, mind if I ask how you did it?
Hi yes, my solution works okay. It doesn’t give 100% uniform results, but it helps.
Declaration:
UFUNCTION(BlueprintPure)
static float ExpFadeOutSineWave(float time, float exponent, float frequency, float startValue, float amplitude);
Definition:
bool UCustomBlueprintFunctionLibrary::FindCollisionUVFromHit(const struct FHitResult& Hit, FVector2f& UV, int32 UVChannel, float& stretch)
{
USkeletalMeshComponent* Mesh = Cast<USkeletalMeshComponent>(Hit.Component);
if (Mesh) {
FSkeletalMeshRenderData* RenderData = Mesh->GetSkeletalMeshRenderData();
FSkeletalMeshLODRenderData& LODData = RenderData->LODRenderData[0];
FRawStaticIndexBuffer16or32Interface* Indices = RenderData->LODRenderData[0].MultiSizeIndexContainer.GetIndexBuffer();
if (Hit.FaceIndex * 3 + 2 > Indices->Num()) return false;
const int32 Index0 = Indices->Get(Hit.FaceIndex * 3 + 0);
const int32 Index1 = Indices->Get(Hit.FaceIndex * 3 + 1);
const int32 Index2 = Indices->Get(Hit.FaceIndex * 3 + 2);
const FVector3f Pos0 = Mesh->GetSkinnedVertexPosition(Mesh, Index0, LODData, RenderData->LODRenderData[0].SkinWeightVertexBuffer);
const FVector3f Pos1 = Mesh->GetSkinnedVertexPosition(Mesh, Index1, LODData, RenderData->LODRenderData[0].SkinWeightVertexBuffer);
const FVector3f Pos2 = Mesh->GetSkinnedVertexPosition(Mesh, Index2, LODData, RenderData->LODRenderData[0].SkinWeightVertexBuffer);
FVector tPos0;
FVector tPos1;
FVector tPos2;
tPos0.X = Pos0.X;
tPos0.Y = Pos0.Y;
tPos0.Z = Pos0.Z;
tPos1.X = Pos1.X;
tPos1.Y = Pos1.Y;
tPos1.Z = Pos1.Z;
tPos2.X = Pos2.X;
tPos2.Y = Pos2.Y;
tPos2.Z = Pos2.Z;
const FVector2D UV0 = Mesh->GetVertexUV(Index0, UVChannel);
const FVector2D UV1 = Mesh->GetVertexUV(Index1, UVChannel);
const FVector2D UV2 = Mesh->GetVertexUV(Index2, UVChannel);
/*Calculate the UV stretch*/
//Calculate area of 3D triangle
// Calculate two vectors forming the sides of the triangle
FVector3f side1_3d = Pos1 - Pos0;
FVector3f side2_3d = Pos2 - Pos0;
// Calculate the cross product of the two vectors
FVector3f crossProduct_3d = FVector3f::CrossProduct(side1_3d, side2_3d);
// Calculate the magnitude of the cross product
float area3D = 0.5f * crossProduct_3d.Size();
//Calculate area of 2D triangle
// Calculate two vectors forming the sides of the triangle
FVector2D side1_2d = UV1 - UV0;
FVector2D side2_2d = UV2 - UV0;
// Calculate the cross product (which results in a scalar in 2D)
float crossProduct_2d = FVector2D::CrossProduct(side1_2d, side2_2d);
// Calculate the absolute value of the cross product and divide by 2 to get the area
float area2D = FMath::Abs(crossProduct_2d) * 0.5f;
// Calculate stretch
stretch = sqrt(area2D) / sqrt(area3D);
const FVector LocalHitPos = Mesh->GetComponentToWorld().InverseTransformPosition(Hit.Location);
const FVector BaryCoords = FMath::ComputeBaryCentric2D(LocalHitPos, tPos0, tPos1, tPos2);
FVector2D tUV = (BaryCoords.X * UV0) + (BaryCoords.Y * UV1) + (BaryCoords.Z * UV2);
UV.X = tUV.X;
UV.Y = tUV.Y;
return true;
}
else {
return false;
}
}```I am distributing particles from Niagara onto a skeletal mesh using the skeletal mesh location module, densely packed. Alternatively, they can be arranged in a grid to optimize their number. I gather the locations of points from which I want to retrieve UV coordinates, in my case, the hit points on the skeletal mesh capsules. I find the particle closest to this point and then use the “traverse skeletal mesh” function to move this point to the exact location for sampling. This way, we have a single particle with a Triangle ID and barycentric coordinates. With the Triangle ID and barycentric coordinates, we can sample the UVs using the “Get Tri UV” function and potentially export them as two floats back to the blueprint. Everything can be processed within one frame, and the systems can be removed in the next frame.
Something like this: https://youtu.be/alQEf454PjU?si=5vGOWTjERr1i4nYM&t=1245
Very elegant. But it seems a little overcomplicated compared to the methods above, which just need complex collisions to work. Maybe I am missing some major advantages of the Niagara method.