I was struggling with the same issue. I also needed morph targets to be supported alongside the dual quaternion skinning. I was able to fix it by creating a deformer graph with a custom kernel:
The custom kernel is configured like this:
Be sure to also include the DualQuaternion functions in Additional Sources. They ship with the engine and the kernel won’t work without. Finally, put the following code in the Shader Text field:
KERNEL
{
if (Index >= ReadNumThreads().x) return;
float3 LocalPosition = ReadPosition(Index);
float4 LocalTangentX = ReadTangentX(Index);
float4 LocalTangentZ = ReadTangentZ(Index);
float3 DeltaPosition = ReadDeltaPosition(Index);
float3 DeltaNormal = ReadDeltaNormal(Index);
LocalPosition = LocalPosition + DeltaPosition;
LocalTangentZ.xyz = LocalTangentZ.xyz + DeltaNormal;
float2x4 DQuat = 0;
int NumBones = ReadNumBones(Index);
for (int BoneIndex = 0; BoneIndex < NumBones; ++BoneIndex)
{
float BoneWeight = ReadBoneWeight(Index, BoneIndex);
float3x4 BoneMatrix34 = ReadBoneMatrix(Index, BoneIndex);
float4x4 BoneMatrix44 = transpose(float4x4(BoneMatrix34, float4(0, 0, 0, 1)));
float2x4 BoneDQuat = DualQuatFromMatrix(BoneMatrix44);
if (dot(BoneDQuat[0], DQuat[0]) < 0)
{
BoneDQuat = -BoneDQuat;
}
DQuat += BoneWeight * BoneDQuat;
}
DQuat = DualQuatNormalize(DQuat);
float3 SkinnedPosition = DualQuatTransformVector(DQuat, LocalPosition);
float4 SkinnedTangentX = float4(normalize(DualQuatRotateVector(DQuat, LocalTangentX.xyz)), LocalTangentX.w);
float4 SkinnedTangentZ = float4(normalize(DualQuatRotateVector(DQuat, LocalTangentZ.xyz)), LocalTangentZ.w);
WriteOutPosition(Index, SkinnedPosition);
WriteOutTangentX(Index, SkinnedTangentX);
WriteOutTangentZ(Index, SkinnedTangentZ);
}
This is essentially the code in the DualQuatSkinning node that ships with the engine, just slightly extended to first manipulate the vertices as required to support the morph target. Note that DQS in general does not support scaled bones, so if any of your bones are scaled, you will have strange results.
Disclaimer: I did not test this extensively. My morph targets only make small adjustments to the mesh. I don’t know how this performs with bigger adjustments, so your mileage may vary.