Retarget Input Pose not working as expected

Development Character & Animation
, ,
1

I’m having issues with the “Retarget Input Pose using Input Pose” setup not working as expected on my MetaHuman characters.

My goal is to have the hands match the UE5 Mannequin pose. When I use “Retarget Pose From Mesh” in the Anim Blueprint—referencing a skeletal mesh component with the Mannequin mesh as the source—it works correctly. However, we don’t want to keep an additional Manny mesh in the scene or character just for that purpose.

It seems that the ABP node cannot retarget properly from an input pose—in my case, the hands end up offset. Could you confirm whether this is a known limitation or if I’m missing a setup step?

For reference, please check the repro project—specifically the characters under the sign/text labeled:

  • “IK Retargeter (Source Mesh)”
  • “IK Retargeter (Input Pose)”

The Source Mesh characters have their hands perfectly aligned with Manny. The Input Pose meshes, especially the female metahuman, has a big offset. Why are both methods not looking the same?

Thanks in advance for looking into this!

Repro Project: https://drive.google.com/file/d/1AtsYFKlVmhgo7u5PxRTc0yeg\_D7pNyhE/view?usp\=sharing

Best regards,

Matthias

[Attachment Removed]

2

Steps to Reproduce

  • start new third person template
  • import/create a metahuman
  • create anim blueprint for metahuman_base_skeleton
  • add a sequence player with an animation for SK_Mannequin
  • add Retarget Pose from Mesh node and change it to Retarget From Source Pose Pin
  • assign a retargeter with hand IK enabled (blend to source 1.0)
    [Attachment Removed]
3

Hi, unfortunately, this is a side effect of the new retarget from source pose pin functionality relying on the compatible skeletons system to generate the input pose that goes into the retargeter.

The compatible skeleton system is what allows us to use an animation sequence in an animation blueprint that targets a different skeleton to the one that the animation was originally authored for. In that setup, when the animation is sampled by the animation player node, the bone data has a pre-transformation applied, which accounts for differences in the ref pose between the source skeleton and the target/compatible skeleton. This is the only way that we’re able to play back animations on different skeletons to the ones that they were authored for and maintain a reasonable pose. If we just played the animation directly on the target skeleton without the compatible skeleton pre-transformation, then the animation would break the proportions/orientations of the target skeleton when applied.

But this does mean that the transforms from the animation have been modified by the compatible skeleton system. So the pose won’t be identical to the pose if the animation is played on the source skeleton. And that’s where the discrepancy in this case originates. It’s not that the retargeter is generating a different pose, it’s that the input pose to the retargeter is different when using the ‘retarget from source pose pin’ functionality (via an animation using compatible skeletons) vs using ‘retarget from source mesh component’.

You can see that to an extent in this screenshot. The overlapping meshes on the left are just playing the same animation. There’s no retarget going on, just compatible skeletons allowing the animation that was authored for Manny to be played on the Metahuman. On the right, we are using the retarget from source pose pin to retarget the animation which is already being played via compatible skeletons on the MH mesh. You can see how on the right, the retargeter maintains the proportional offsets between the different meshes that was introduced by the compatible skeleton system:

[Image Removed]Whereas in this screenshot, I’m retargeting directly from the source Manny mesh to the MH mesh. There’s no compatible skeleton offset being introduced here.

[Image Removed]Unfortunately, we don’t have a solution apart from using the retarget pose from source mesh option, or performing the retarget offline in the editor so that the source animation targets the MH skeleton directly. Long term, our solution for this is to be able to play animations on arbitrary skeletons (with no fixup) which will be possible via the new Unreal Animation Framework. However, it’ll still be some time before it reaches a point where it’s viable for use in production projects.

There are some other options here, but they would require code changes. You could add a flag onto bones in the skeleton to specify whether they should use the compatible skeleton pre-transformation or not (in a similar way to how we have flags for the ue4 style/translational retargeting per-bone). But that wouldn’t be trivial. You could also try just changing the skeleton that the animation sequence targets (‘r-click > assign skeleton’ in the asset browser) but I think that’ll probably break the animation since the Manny and Metahuman skeletons are quite different.

Happy to discuss this further if you’ve got questions.

[Attachment Removed]

4

Just a heads up that I was looking at this again with some repro assets that I set up specifically to have more closely matching ref poses, and I still see a difference between the input pose input to the retargeter vs the source mesh component input. So there may be something more going on here than the compatible skeleton pre-transformation of the bones (and also translational retargeting settings, which also affects the input pose).

I’m going to do some more digging on this, I’ll follow up once I have some findings.

[Attachment Removed]

5

Hello [mention removed]​, thank you for looking into this.

Our specific problem is that the characters need to carry items in their hands, and it looks very different on female metahumans compared to male ones.

Probably the easiest solution is to create separate animations for females, but the same problem might pop up in other scenarios.

It would be amazing if we could use runtime retargeting to always have the female hands in the right place, but having an extra skeletal mesh around for each female to use RetargetPoseFromMesh is also not a viable option.

Do you have a recommendation what to do?

[Attachment Removed]

6

Hi, sorry for the delay following up on this. There’s a lot of moving parts with this problem, so it’s taken quite a while to debug and talk through the problem with the dev team.

Essentially, the retarget from source pin option isn’t going to give you quite what you want here with the current way that the node is implemented. The problem is that we have different systems fighting against each other with translational retargeting, compatible skeletons, and IK retargeting. The retargeter relies on the input transforms being consistent with the source retarget pose. But we don’t have that here because, when you’re doing the retarget within a single anim bp, the animation has already been conformed to the target mesh by translational retargeting and compatible skeletons. So the source retarget pose no longer matches the input transforms.

Ideally, what we would do to get this setup working would be to disable translational retargeting on the target mesh (which we can currently do), and disable compatible skeleton remapping (which isn’t currently supported). If that were possible, when the animation was sampled, it would give us transforms that exactly match the source mesh. We could then pass that to the retargeter and have it perform the retarget. There is another limitation there in that the retargeter currently only have access to the target retarget pose and not the source retarget pose when performing the retarget inline like this, so we’d have to add another option to the retarget node to specify the mesh that the source retarget pose should come from.

In terms of solutions in the short term though, you have a few options:

  • Retarget the animations offline so that you have a set of animations that have been correctly retargeted from the Manny mesh to the female MH mesh.
  • Don’t use IK retargeting and just use translational retargeting instead. From testing with the source animation that you sent over, translational retargeting gives a closer result to the input pose. In the example below, the green skeleton is the MH mesh, just playing the holding animation (with translational retargeting changing the proportions of the bones) and the yellow skeleton is the same animation on Manny. I found I got the best results with this when I set all of the lower body and the pelvis to spine to use ‘Animation Scaled’, and I set the chain from the clavicles down to use ‘Animation’ (see the second screenshot). The issue with this approach is that while it works for this pose, it may not work for others. But it’s worth testing. [Image Removed]
  • The last option is to customize the retarget node like I described so that it’s possible to specify a source mesh to take the retarget pose to retarget against from. This should work if there are no orientation differences in the ref pose between the two meshes (I would need to double check this with the MH vs Manny meshes) and you set the translational retarget settings to Animation for all the bones. If you’re interested in this approach, let me know and I can investigate further.
    [Attachment Removed]
7

Sorry for the delay following up on this, I wanted to take another look at the retargeting setup to see if I could get any better results by reinstating the inline IK retargeting after the changes to the translational retargeting that I mentioned previously. However, that gives the following results:

[Image Removed]And I don’t think that’s much better (if at all) than where you started with the original setup?

> the translational retargeting setup you suggested doesn’t yield the best visual result, as it gives the females male proportions including wide shoulders

This makes sense since one of the main offsets between the rigs is the clavicle and shoulder widths between the MH female vs Manny. So by offseting the female clavicles and shoulders to match the width of Manny, you end up with the hands in the correct location but you then also have the wider shoulder pose. Re-applying the IK retargeting compensates for the clavicle/shoulder pose but then the hands are offset.

> customizing the retarget node like you described sounds good. is this even possible in a plugin, without modifying the engine? if so, we would be very interested in that approach.

To do this without modifying the engine would require duplicating the code for the IK retarget animation node, allowing it to live within your plugin. Then it could be customised to add support for specifying the mesh to take the retarget pose from. That assumes that there are no issues with public/private dependencies (and also that the compatible skeletons system isn’t causing any of the offset). If this kind of duplication and modification of the node is acceptable to you, let me know, and I can investigate whether I can put together some code for you.

[Attachment Removed]

8

Quick follow up on this. I confused myself when I was looking at the setup again today. I was comparing the pose of Manny vs the MH female, which wasn’t the original problem. Your original issue was the pose of the MH female when inline retargeting vs pose of the MH female when retargeting from source mesh. When comparing that setup (using the new translational retarget settings + inline IK retargeting) I get the following results:

[Image Removed]The results between the two MH female meshes here are much closer (there is just a very small offset in the pose of the fingers). And with this approach, the clavicle/shoulder width still retains the correct proportions for the female mesh (thanks to the IK retarget). Does this possibly give you the results that you need?

[Attachment Removed]

9

Sure, in that example, I had the following setup with the translational retarget settings:

root: animation

pelvis -> spine_05: animation scaled

clavicle_l/r -> fingers: animation

thigh_l -> toes: animation scaled

thigh_r -> toes: animation scaled

Essentially, what I was trying to do was to replicate the main proportions of Manny on the MH female (ie. the higher pelvis, which also requires longer legs to compensate) and the wider shoulders.

And the anim bp running on the mesh was the one you set up originally:

[Image Removed]

[Attachment Removed]

10

> I wasn’t aware this was the method that makes the shoulders wider. unfortunately, this is not an option.

Sorry, that’s not quite what I meant. I meant that the transforms sampled from the anim sequence will have wider shoulders since they’ll be using the animated bone lengths (for the clavicles onwards) via the translational retarget settings. But those transforms will then go through the IK retarget process, which should bring them back to the correct proportion for the female MH character. The idea is to make the transforms that go into the IK retarget node, when doing the in-line retarget, as close as possible to those that would be passed to the node when doing the retarget from the source mesh.

Can you give those retarget settings a try on the MH Female mesh? I think that should give pretty much identical results to the retarget from source mesh

[Attachment Removed]

11

Hi, sorry for the delay in following up on this. I’m out on leave at the moment, so I’m going to pass this issue over to one of my colleagues.

However, just before that, I wanted to attach a revised version of the project that I was testing with. When you open the project, it should open Lvl_RetargetTest. This contains the test assets that I’ve modified from the originals that you sent over. The level has two actors in it, one running retarget pose from source mesh (via an actor blueprint containing two meshes). The other is a skeletal mesh actor running the inline retarget via an anim bp (with the translational retarget settings I mentioned previously). I’ve positioned both of the female meshes so that they overlap. With this setup, if you PIE/simulate, the poses of the two female meshes are almost identical.

[Image Removed]Hopefully that helps, but if I’ve missed something, then just let us know.

[Attachment Removed]

12

hello again, thank you for the detailed answer. I have considered your suggestions:

  • retargeting the animations offline is definitely a possiblility, although it would only fix one specific animation at a time.
  • the translational retargeting setup you suggested doesn’t yield the best visual result, as it gives the females male proportions including wide shoulders: [Image Removed]it also might negatively impact other characters using the same skeleton and their animation, such as trolls with ape-like proportions etc., so it sounds dangerous in a large scale production.
  • customizing the retarget node like you described sounds good. is this even possible in a plugin, without modifying the engine? if so, we would be very interested in that approach.
    [Attachment Removed]
13

the version on the last picture doesn’t look bad, could you explain again in detail what the setup for this one was? thanks in advance😊

[Attachment Removed]

14

sorry, I wasn’t aware this was the method that makes the shoulders wider. unfortunately, this is not an option.

I would like to come back to the customization of the retargeting node, as I feel like this is our only option:

“If this kind of duplication and modification of the node is acceptable to you, let me know, and I can investigate whether I can put together some code for you.”

Some duplication would be acceptable, and we would very much appreciate if you could assist with that.

[Attachment Removed]

15

just to get it right:

now have set the retargeting on the female as you described, and she gets wide shoulders if I just play the animation without the ABP.

[Image Removed]

the ABP version however has not changed for me at all, am I missing another step? (game was simulating)

[Image Removed] [Image Removed]

[Attachment Removed]