Fixed Single Camera Calibration for MetaHuman

I’m using a FaceWare Mark IV system and the only option I’m seeing for calibration is the single neutral pose option.

I’ve been getting less than ideal results on the lips and forehead.
The forehead especially so, it tends to move a lot when talent is blinking their eyes. The lips don’t always come together during speech. The lower jaw movement is also extremely limited.

If anyone has suggestions, guidelines, best practices of using the Mark IV system for live Performance Capture. As well as even using MetaHuman as a post processing system for HMC Data that may not have a stereo camera system that doesn’t include a left, right facing pose.

Cheers!

Hi @PaulThibodeau | Welcome to the community and thanks for bringing this up.

Getting poor solver performance on the mouth, brow and jaw when using single camera/mono is very frustrating when you get blinking and talking issues.

Before diving into a few workflow adjustments we could try, Could you share a bit more context?

  • Which Unreal Engine version are you running?

  • Are you piping this through MetaHuman Animator using a mono capture setup or handling the FaceWare Mark IV data via an external solver/retargeter before bringing it into UE?

In most cases you will be limited to having only one calibration option for a neutral pose (you are using mono/single camera and not using stereo rigs or calibration boards)
This means that you won’t have enough stereoscopic depth data for the solver to accurately calculate the small translations on the jaw and upper face.

If you are using MetaHuman Animator for mono video please refer to MetaHuman Animator from Mono Video Capture Documentation
If you are using external HMC data within standard pipelines then it is vital that your neutral frame has absolutely zero tension in expression.

Once you let me know these additional details then I can help you find the best way to clean up the tracking data.

  • Which Unreal Engine version are you running? - Currently testing on 5.8 and 5.3

    Are you piping this through MetaHuman Animator using a mono capture setup or handling the FaceWare Mark IV data via an external solver/retargeter before bringing it into UE? - I’m connecting the video feed in via LiveLink —>MetaHuman (Video) —> Device Elgato

    The video is coming direct from Hyperdeck 4k studio pro where the raw source could be recorded (over HDMI so in the future I can bring in audio)

    This is what the whole line looks like
    Talent —> FaceWare Mark IV (mono camera) —> Hyperdeck —> Elgato —> PC —> LiveLink —>MetaHuman (Video) —> Device, Elgato

It is a mono camera setup, I have not used a calibration board, my understanding was the calibration board method was for dual camera setups.

I did use the capture neutral option.

When desired we are bringing in body and head motion from a OptiTrack system. (For the testing at this point it wasn’t used)

I’ll give the MetaHuman Animator for Mono video a check, thanks for sharing that one.

Hey there! Thanks for providing this info and screenshots!

In MetaHuman Animator when using monocular set-ups (without using a stereo calibration board for 2 cameras) you have to rely heavily on accurate depth approximation of the solver. Issues that often occur are small brow/forehead twitches (particularly when blinking) and a lack of lower jaw movement.

These are a few changes I would try to help improve the forehead stability, lip closure and range of jaw motion

1. Fine-Tune MetaHuman Identity & Neutral Frame Selection

  • Zero Expression Tension
    Ensure the frame used for Capture Neutrals is taken when the talent facial muscles are completely relaxed. Any micro-tension around the eyes or mouth during neutral capture will offset the baseline neutral pose, causing blinks to incorrectly pull the forehead mesh.

  • Re-run Curve/Pose Tracking Alignment
    In your MetaHuman Identity asset, ensure the tracking markers around the lips and eyebrows are manually adjusted on the neutral frame before running the solve

2. Enable Control Overrides in MetaHuman LiveLink/Solver

In your Capture/LiveLink settings window shown in your screenshot:

  • Head Translation/Stabilization
    Since you plan to bring in head and body motion from OptiTrack later, keep Head Translation and Head Stabilization disabled to prevent the facial solver from introducing unwanted secondary mesh rotation/translation artifacts.

  • Focal Length adjustment
    Make sure the Focal Length value (currently set to 2223.77 px in your settings) accurately matches your capture lens field of view
    An incorrect focal length value forces the monocular depth solver to distort scale which directly impacts jaw opening depth and lip-sync accuracy.

3. Capture & Lighting Best Practices for Mono HMC

  • Diffused Neutral Lighting
    Single-camera solves depend heavily on shadow intensity to estimate depth. Avoid harsh top-down lighting on the head rig which creates shadow movement when eyes blink.

  • Unobstructed Eyelid and Lip Margins
    Ensure the inner lip seal line and upper eyelid contours are completely visible and clear of reflections or stray hair.

Epic Documentation: Video Capture Guidelines · Generate Calibration

Give those neutral pose and focal length tweaks a shot and let me know if the forehead jitter stabilizes and if it helped you!

Want to say thank you and let you know I’ll be looking into these suggestions and tips.
However, I won’t get the chance until the end of next week I’ll follow up with results then.

Thanks again and cheers!

Thanks for your question and to @BRS_GerDAngelo for providing some steps to help improve the animation quality. Animation from a single camera through Live Link doesn’t use a MetaHuman Identity asset however, so I wanted to clarify a few points before you get back to testing @PaulThibodeau

The MetaHuman (Video) Live Link source works out the facial animation directly from the camera image. There is no Identity asset and no depth data involved in the facial animation, so there are no tracking markers to adjust.

Focal Length is a read-only field. It is estimated automatically when you press Capture Neutrals, so the 2223.77 px in your screenshot is that estimate. It is only used for head position and head stabilization and does not feed into the facial animation, so there is nothing to match to your lens.

The advice above about a relaxed neutral is the important one. With the default settings, when you press Capture Neutrals, the values being produced for the face at that moment become the zero point for the brows, eyelids, cheeks, nose, mouth and jaw. Later frames are measured from that point and rescaled, more strongly the further from zero the captured values were. Nothing checks that the face is actually relaxed, and the capture happens a few frames after the press, while the button shows “Hold neutral pose”. Any tension or a slightly open jaw at that moment gets baked into the baseline. I can’t tell from here whether that happened in your case, but it is quick to redo: face as relaxed as possible, not mid-blink, and hold it until the button returns to normal.

One thing that could improve your results is adjusting the Smoothing configuration in the same subject settings. It is applied on a per control basis. If the brows are still unsteady after a clean neutral, you can assign your own smoothing asset with more smoothing on the brow controls.

I can see you already have Head Orientation, Head Translation and Head Stabilization turned off. Those only affect head movement, not the facial animation, so that is fine when the head is coming from OptiTrack.

Since the facial animation comes entirely from the image, the tips above on lighting and keeping the lips and eyelids clearly visible are still worth following.

Setting Monitor Image to Trackers overlays the tracking on your video, which is useful for judging how stable the tracking is.

For recorded footage, the MetaHuman Performance asset has a mono footage option that uses the same solver on single-camera video, with its own processing settings. It is worth knowing that a mono solve is not a direct match for the offline depth processing pipeline (stereo HMC or Live Link Face footage), which will provide a higher fidelity result.

Let us know how you get on!