Hi @nwell26 I’ve been looking into your problem and trying to replicate it on my end to see if I can find a work around for you. Although I was not able to replicate your problem “the body animation’s arms/hands stop animating correctly” I was able to replicate something that I think is close that may help you. In the process I will go over with you we will go over the setting I have for Live link, the settings I’m using to process for the Metahuman performance and how to bake the animation we just processed into it’s corresponding control rig so that if we need to fix or teak some of the animations we can do it there instead of re-recording your video reference and waiting for it to process.
In the following information I’m about to provide I’m assuming a few things:
- you already know the workflow for assembling a metahuman character
- how to set up your animation sequence
- the videos you are using to capture the mocap data are well lit and capture the subject you are trying to extract mocap data from (example if you are using one this trying to capture the body you have a video that shows the full body)
- you are using one video to capture the body and a separate video to capture the facial performance
- you have synched both videos to match the audio and overall animation output either in editor or in your preferred video editing program that way both videos match their timing (if you haven’t you can do that in the sequence itself later on but may require more clean up later)
Okay with that said let’s pick up in editor (there are steps that you may have already done or know how to do that I will go over. Just in case, I will label them so if you need to you can just skip it and find the information you need, the reason I am adding it is in case you run into a problem you could double check with my previous instructions to see if you missed something or need to tweak anything):
Metahuman Assembly:
I created my metahuman character, I removed the clothing for now so that I can see better which parts of the body break during the process, I named it, picked the folder in which I would like the information to be assembled in and clicked assemble.
Setting up the File structure and scene:
Let’s navigate to the folder where I assembled the metahuman in and add it to my level. Additionally, I have created a few more folders in there to keep things organized. The folders are:
- rawAnim: this is where I will process the raw mocap animation from my footage
- Mocap: is where my Metahuman Performance files are located
- Anim: is where I will assemble everything and test
Notes:
- If you notice my blueprint at the moment is not saved this will come back to bite us in the future, I’m leaving it unsaved just in case this happens to you in the future so that you know how to fix it the issue we will run into
.
Setting up the Videos in Live Link:
Lets go to Tools → select Ingest Server
Then let’s open up Tools again → open Live Link Hub
With the “Live Link Hub” window open please navigate to “Live Data” and select “Capture Manager” from the drop-down menu.
Let’s add a Mono Video Digest by clicking on the “Add” button right underneath where you were and selecting the corresponding option.
navigate to the folder where your videos are located (you will see all of the videos in that folder populate your middle window) → name it
Select the videos you would like to use and click “Add to Queue”
In my case the video I took was set to portrait mode so I will have to apply a rotation to it before I process it. So I’m selecting both and going to “output” → Video → Image Rotation → Clockwise 270
Select Start to start processing
you can find the processed data in your Content → Capture Manager, there you will see a folder for each individual video processed
MetaHuman Performance:
I will navigate to my Mocap folder that I had previously created and create 2 Performance files, one for the Body and one for the Face captures.
Mocap folder → right click inside the content browser → MetaHuman → MetaHuman Performance → name them accordingly
Let’s open up the body capture one (both of them are empty at the moment since we just created them)
In the “Data” panel we are assigning the footage we just processed earlier
In the “Visualization” panel we are assigning the metahuman preview
Okay let’s assign the processed data for the body capture
Details → Data → None → Select it to bring up the drop-down menu and select the Body capture data we processed.
In the details panel let’s scroll down a bit, let’s make sure that:
- Show skeleton is enabled
- We are going to deselect “facial tracking” since we just want the body capture information not the face (don’t worry this will still capture the rotation of your neck)
- and Select “Body Tracking”
Note:
- you can leave auto height selected or unselect it and apply your heigh in centimeters
- If you add a model to the visualization before you process, based on my experience it takes longer to process, that’s why I recommend doing after at least for the body.
Let’s get to processing by selecting the “Process” button and give it time to go thru it (this will take some time depending on your hardware set up and the length of your video)
Once it’s done Processing, you can now begin to scrub thru the timeline and see how your skeleton is tracking based on the video you provided. We can now select the metahuman we made and assign it to see how it looks in the preview, keep in mind that the results you get will be heavily based on the quality of your video, lighting, camera position, and also keep in mind that this is still under “experimental” so do not expect to get 100% accuracy out of it.
Visualization → None → look for your Metahuman blueprint
If you can’t find it like it happened to me please make sure that the MetaHuman blueprint you assembled at the start of the project is saved (this is the problem I was refereeing to earlier). Additionally next to the search bar if you click on the gear, you can now see the folders it is letting you search from so make sure the folder where your blueprint is selected.
If when you assign the metahuman BP to the visualization tab and the pose does not update please scrub thru the timeline, if the skeleton is the only one animating, please save all that you have done in the project and restart the editor, if you do not this could cause unreal to crash. Once you restart the editor it should work.
Now that we have confirmed that the character is animating. Let’s go ahead and export the animation by going to the “Export Animation” button → changing the settings accordingly if needed (I didn’t have to change it) → click on “Create” and place it in the folder you would like to store it in I am using the “rawAnim” folder I created earlier.
Lets do the same for the face but change a few settings. We’ll import the face capture from the facial data we captured and select the metahuman head we would like to work with.
We will only be using facial tracking for this so we will keep facial tracking on and keep body tracking off. in the Facial tracking section please navigate to Focal Length and click on “Estimate”, and you should be good to go to start processing
note: the processing of the face should be faster than the one for the body.
Once you have it processed and you can confirm that the metahuman head you brough in is animating correctly let’s go ahead and export.
This is important if we export the animation with “Enable Head Movement” it will break. You might come across the issue that your body and head are now detached once you combine the animations, so we want to make sure you export with it turned off.
Save it in the folder you would like to use moving forward (I’m still using rawAnim folder)
Testing the animations
Create a Level Sequence and name it, I will be using this sequence to test the animations.
Before we add the blueprint to the sequence please select your Metahuman blueprint in the level → navigate to details panel → select the body → go down to “Animation Class” → Set it to Use Custom Mode
Do the same for the face and drag and drop the blueprint from the outline to the sequence.
Mute the control rigs by clicking on the eye icon for the body and face for now and add the animations for the body and face capture, (click on the “+” sign next to body and select animation, select the body animation, go ahead and do the same for the face)
Scrub thru the timeline and you should see both animations playing without breaking, in my case I can see that that the shoulders of my metahuman and drooping, an easy fix for this is to bake the animation we have on the body into the “Body Control Rig” and tweak it adding an animation layer.
Right click on Body → Bake to Control Rig → deselect filter asset by skeleton → close the window and open it again or else that change will not take place and search for the control rig for your Metahuman → and create
since all I want to fix for now is that shoulders I will select the left and right clavicle controls and assign it an animation layer
With the clavicles selected I will open their attributes by clicking on the triangle next to the text and scroll down to find the Rotation.Pitch and assign for each a value of 10, I will also scrub thru the animation to see where the shoulder break and assign the animation keys by clicking on the Dimond with the “+” sign and changing the rotation accordingly based on the animation.
and that should fix the shoulders
I hope this helps, please let me know if you come across any problems or need any further assistance. Good luck with your project .
