See above. Python-based capture data asset creation (video data, face recordings) produces broken depth maps.
Such broken data does not alow me to automate the processing any further (MH identity creation, MH performance creation).
Steps to Reproduce
I’m trying to automate face capture video data processing to Unreal Engine, to Metahuman Identity & Performance, in python, accordingly:
- CaptureData asset, from “top” & “bot” folders (two cameras), for Metahuman Identity creation
- CaptureData asset, from “top” & “bot” folders (two cameras), for Metahuman Performance animation
- Metahuman Identity asset creation (Face with key frames, Identity solve, Prepare for performance)
- Metahuman Performance processing (animating the created MH Identity skeletal mesh)
This is supposed to be python implementation, drawing from this Epic example:
UE_5.6\Engine\Plugins\MetaHuman\MetaHumanAnimator\Content\Python
The issue: when automating Capture Data import using Python script, the created depth processing data (EXR) is always faulty.
As in: face not registered at all in the image, the spatial chunks identified are way too large.
This is not source data issue, as when I create Capute Data from GUI-based import in UE (create Capture Asset, process in Capture Manager), the resulting UE-generated depth processing EXR files are alright.
This is better illustrated on actual depth images. This depth image is created from GUI interface and it processes OK:
[Image Removed]
Whereas this one is created through python pipeline (broken, no face detected):
[Image Removed]
As MH Identity creation process fails to detect face in depth data, I cannot automate the process any further.
Care to investigate or explain what is going on there?
---
For further reference, here are capture data import settings when processing in GUI:
[Image Removed]
Capture data import settings when processing through python:
UE_5.6\Engine\Plugins\MetaHuman\MetaHumanAnimator\Content\Python\create_capture_data.py, function import_take_data_for_specified_device()
I’ve also looked into the python class, editing the existing code by trying to replicate the GUI settings, accordingly (see screenshot, added lines 140-144):
[Image Removed]
However, no matter what settings i set there for depth processing, the python-created depth image data is still broken.
Also attaching my MH Identity automation script as attachment (quasi-working, import_take_data_for_specified_device() is called in the main run() function, line 333).
Hi Pavel,
Thank you for providing the detailed information.
Poor depth data is usually a sign of poor calibration, and so my first thought is that there is some misalignment between the calibration used when generating the identity through the Python script compared to the UI. Additionally, it looks like you are using Unreal Engine 5.6 and there have been some workflow changes here since 5.5 so I want to make sure that is also not contributing to the problem.
When solving performance from stereo camera pairs in Unreal Engine 5.5, the workflow was to use the Stereo Camera Calibration tools to prepare the data for ingest. An early step in this process would have been creating the calibration from a checkerboard take, which is then associated with identity and performance takes via the take.json file. The processed takes are used with Capture Manager for ingest, and it is ingest that generates the depth data ahead of identity creation and performance processing.
From Unreal Engine 5.6, the workflow changes slightly. Ingest is now done directly using Capture Manager in Live Link Hub, and without any pre-processing using the Stereo Camera Calibration tools. Once ingested, calibration is generated in Unreal Engine from the relevant Capture Data asset. After calibration, depth is also generated in Unreal Engine, and must be done for each identity and performance take. Then the identity creation workflow picks up as before. There is some documentation available for the Python scripts that accompany this updated workflow.
While the ‘old’ Capture Manager tool still exists, and should work, I think the first thing to try is ensuring you get a good depth result using this new workflow. Are you able to test that?
Also, I can’t obviously spot the calibration and depth generation in the Python script you shared, although I can see references to those files. How are you doing that at the moment?
Mark.
Hi Mark,
Well … the situation is still very much unclear, still bugged out for me the same.
Regarding: “Poor depth data is usually a sign of poor calibration, and so my first thought is that there is some misalignment between the calibration used when generating the identity through the Python script compared to the UI”
I am using the same set of data, per the two calibration screeshot in the original post. In the first screenshot, the depthmap is processed OK in the GUI of the “old” Capture Manager (from a Capture Source asset); the second screenshot shows garbled depthmap as processed by python scripting. So, workflow issue.
As for initial data:
- I’ve got top/bot folders with per-camera image recordings
- I also have an existing camera calibration ready, precomputed elsewhere (see attached “lensCal” JSON file)
As per your comment: " Once ingested, calibration is generated in Unreal Engine from the relevant Capture Data asset."
That workflow does not make sense to me (?)
I have one video capture where the calibration board ( https://dev.epicgames.com/documentation/en-us/metahuman/calibration-takes?application_version=5.6 ) is waved in front of the camera. Other video captures are human actors only, with no calibration board waving (so I don’t see how I’d be computing calibrations there).
Per your reference to the UE 5.6 depth_generator_example.py script, I tried to run it on existing data I have prepared, Like this:
D:\UE_5.6\Engine\Plugins\MetaHuman\MetaHumanAnimator\Content\DepthGenerator\depth_generator_example.py --cd-package-path /Game/Faces/Identity/F_NadiaA/MetaHumanCaptureSourceSync_0_Ingested/20250716_1338_1-CD.20250716_1338_1-CD
--> the result of that is the same garbled data, as per screenshot above in the original post (“python pipeline (broken, no face detected)”)
-----
Intention per my process_mh_identities.py script (re-attaching a newer version):
- lines 371-382: create Capture Data UE asset
- Capture Data is created from Epic’s own UE 5.6 script (UE_5.6\Engine\Plugins\MetaHuman\MetaHumanAnimator\Content\Python\create_capture_data.py)
- line 387 to function body (line 207+): create MH Identity -- which crashes, because Depthmap data is wrong (as above)
+ adding calibration file example
Thanks for the updates, Pavel. I am sorry for the confusion (and am not sure what triggered the issue to be automatically closed).
To clarify the workflow (and my earlier comment about generated calibration) the expected workflow in Unreal Engine 5.6 is;
- Ingest a calibration take, a take for the identity, and one or more performance takes,
- Generate calibration data, for the calibration take, from the right click context menu in Unreal Engine,
- Generate depth data for the identity take and performance take(s), using the above calibration,
- Go on to create the identity and solve the performance,
From your updates it sounds like point 1 is correct, however you go on to say the calibration is ‘precomputed elsewhere’. What does that mean - how has it been created?
My initial thought was that there is some mixing of the Python scripts and workflow between Unreal Engine 5.5 and 5.6, where there has been some workflow changes, and I couldn’t obviously see the calibration being referenced in your scripts. Specifically, 5.6\Engine\Plugins\MetaHuman\MetaHumanAnimator\Content\Python\create_capture_data.py has been replaced by the scripts mentioned here.
Which stereo HMC device are you using?
In the meantime, I’m going to need a bit longer to piece together what you are seeing and identify where the issue might be coming in.
Mark.