We’ve got a MetaHuman pipeline set up to automate processing of HMC data, and things are generally running fine, but we have a handful of takes that are problematic. For these takes, the camera rectification code (RectifyCameras function in MetaHumanDepthProcessing/Source/MetaHumanMeshTracker/Pirvate/reconstruction/include/reconstruction/Rectification.h) is returning new camera objects with very large dimensions, so the allocations that use those dimensions are failing. I’ve tried manually modifying the calling code to use the keepImageDimensions option, but that threw an error late in the process “cannot estimate the rigid alignment as there are not sufficient triangulated landmarks (only 0)”. Making some assumptions based on that error, we tried to manually adjust the brightness of the beginning of the videos, but that didn’t help.
I’ve attached the HMC files for a short take that we’re having this issue with, but there are several more that we have as well.
This is using Unreal 5.6.1, and using the MetaHumanDepthProcessing plugin available on FAB. Let me know what else would be useful context to know and I’ll do my best to provide it.
I will take a look at your data today. Generally, if this situation occurs, it means that the camera calibration is ‘bad’, leading to crazy image rectification. Let me take a look and see if I can confirm that and/or find a possible work-around.
Would it be possible for you to send me the calib.json which you used to ingest this shot? The camera calibration footage which you used to generate it would also be useful if that is possible. I need to check whether the camera calibration matches the footage, as that is almost certainly where the problem is.
Got the files, but those look like the Identity footage rather than the camera calibration footage (of the checkerboard pattern). Could you send me those?
Success. I have re-tested using the latest camera calibration tooling now in UE 5.7, using automatic frame selection, and that is giving much better results. See the depth map screenshot below. Did not need bundle adjustment in the end.
I have attached the camera calibration file below so you can re-ingest into 5.6.1.
I have two suggestions going forward:
looking at the camera calibration footage again, it looks like there is not a huge amount of variation in depth / tilt of the camera calibration board. It would be worth increasing this somewhat. I suspect that the default frame selection used in the Camera Calib tools for UE5.6.1 (which just selects every nth frame) is not giving sufficient variation in depth and the camera calibration solution is somewhat degenerate / unstable.
the camera calibration tooling in UE5.7 is much improved in terms of selecting the best frames from a calibration shot, and giving a better calibration. It is somewhat fiddly to get the camera calibration out of UE5.7 and back in a form which can be ingested into UE5.6.1, but it is possible using a secret debugging CVar. If you are seeing this quite a lot, I can make a video showing how to do that or get on a call to walk you through the process.
Let me know if this seems to help at your end to process this shot.
Right, I have processed the calibration. The calibration footage looks good, the resulting reprojection error looks good, but the calibration does not match the performance shot.
There are a number of possible causes for this, but the main one is that the calibration parameters have changed between the calibration shot and the performance shot. This could be due to one of the cameras being refocused, a knock to the stereo pair etc etc.
I assume that that is the only possible camera calibration shot to try?
I have one more thing which I will try: we have an experimental feature in R&D which may be able to ‘heal’ the camera calibration through bundle adjustment. It will likely be tomorrow before I can try this.
Yeah, unfortunately that is the only calibration shot that’s available. It sounds like it might be helpful to have a grid take prior to the cameras being put on the performer, as well as after they’re taken off? That way at least there’s another option that might account for things like this.
Thanks for looking into this so quickly. We have quite a few other takes that are suffering from this problem, so it would be helpful to know how you went about determining that this was the problem. What specifically do you mean when you say “the calibration does not match the performance shot”?
Also, in chatting with one of our facial animators, we do currently take a grid take at the beginning and the end of the day, but we could do so more frequently. Additional comments:
Camera positioning is constantly shifting, but we carefully adjust each actor’s HMC before every select to match the positioning from the DNA take. We never change the focus after the initial framing and setup, but the cameras will always fluctuate by micro degrees-there’s no way to completely avoid that.
The grid take that I uploaded was the first AM take, but on this day we did an additional AM take, as well as a PM take. I’ve uploaded all of those onto the ticket now, so perhaps one of those is a better match. However, that would only help in this case, and it looks like there’s something systemic going on here, as we’re seeing this problem across many takes from many days.
That’s great! Glad you were finally able to coax something usable out of the system.
So if I’m understanding things, the critical difference here was the automatic frame selection that’s in 5.7, as opposed to the every nth frame selection that’s in 5.6? Unless I’m mistaken, at least in 5.6 I believe that is part of the CalibrationApp that’s in the StereoCaptureTools plugin, right? Is that something we could upgrade individually, or has that been brought into the engine in the MetaHumanCalibrationProcessing plugin in 5.7?
Either way, we are indeed seeing this in over a dozen takes so far, so knowing the ins and outs of how you diagnosed it, how you did what you did to fix it in 5.7, and how to pull it back to 5.6.1 would be fantastic.
Correct. The StereoCaptureTools of 5.6 have been brought into engine for 5.7, and there is added functionality to better select the calibration frames, as well as a new experimental plugin which can help diagnose that there is an issue with the camera calibration.
The TL;DR for fixing it was:
Ingest the camera calibration shot into UE5.7
Perform camera calibration in UE using auto frame selection
Perform fiddly workflow to export the camera calibration as a .json file (not a supported workflow but possible via a somewhat hidden debugging CVar)
Use the calib.json file as usual to ingest the performance shot in 5.6.1
Happy to go through that with you on a call if that would help. What timezone are you in? I am in the UK and am unavailable from 5pm onwards today GMT, but can do most times tomorrow. In the meanwhile, I will make a short video showing how to ingest, calibrate and export the .json file in UE5.7 and try and share that with you.