I’m trying to find resources to map some pose data we get from Mediapipe (google’s AI solution that uses openCV to compute skeletal landmarks from an image) to the unreal character (possibly using a control rig with FABRIK). I’ve found little in terms of literature in this regard and I would like for some help from anyone with any tips on how to achieve this. Looking to use camera input and the pose detected from the camera to drive the unreal character so in a sense, the backend tech will be a “controller”.
The above image details the data received. Some considerations are to generate a lower LOD skeletal mesh so the FABRIK solver has fewer bones to solve (ideally 1:1 parity with the image). Expectations are to come up with something that is somewhat responsive since we already have close to 200ms lag from the “backend” where we obtain the above mentioned pose data that we try to fit to an unreal rig. Our current solution uses a full body IK, it’s glitchy, has problems and nowhere near perfect but it is capable of moving the character albeit at a very slow rate (500ms). This rate is found to be unacceptable as it feels sluggish. Any input is appreciated. Also, if anyone can point me to how to use 3D real world coordinate data (x,y,z) points without any rotation information (or perhaps infer the rotation from the point data itself) with FABRIK, it would be highly appreciated. Thanks in advance
Fabrik allows for a pole system to determine rotations.
You should be able to just define a skeleton dependent pole orientation for the solver to automatically compute the rest.
As far as your other points go, computation is expensive. You can limit the computation options on the fabrik node to gain some performance, but it really won’t do all that much.
The biggest problem you have, really, is that you want to use Unreal. An engine that puts 0 importance on performance.
On top of that, IK is solved in real time, so you are CPU bound to however many calculations are required to get a solution…
Try ccdik or some of the less expensive solvers - but usually they do not compute the rotation via pole vectors.
You can likely deterministically pre-compute rotations outside of the engine and provide them to the engine already solved.
It depends on how many MS the math adds to your lag.
Regardless, whether you add it before or after the end result won’t really change. It’s bound to lag behind a bit before the bone position is matched. Avoiding redundant calculations is likely the best way to simplify it…
We once tried to use IK to solve this problem, but the result was not very good. The main problem is that the 3D skeleton position is not accurate and the length of the bones between frames may change, making it difficult to map them to a skeletal mesh for IK.
We also have made a solution using MediaPipe, usage video below,
Appreciate your responses. We already have plans for a 2 camera solution for better pose landmark estimation and we are looking to pivot to FK after your input thank you very much for this. Having said that, what do you guys think about making models that are somewhat “stretchable” to fit the proportions. Does that even make sense logically ? ex: when you turn in front of the camera the pose is detected with very small shoulder to shoulder length since you are standing at a perpendicular angle to the camera plane?
In which case we should be focusing on rotation rather than stretching the shoulders themselves. I’m sure there could be some rotation aspect that could be used to “know” when to resize/stretch and when to turn/rotate. Thanks again for your time
Its cartoonish, but in a pinch it sort of works (unless you use blender to animate, they still have a bug where streching chained bones causes odd rotations of other bones).
If you are going for realism, then stretching is a no. If you simply want somethingn that looks ok and the art direction is somewhat cartoonish, then stretching will do the trick.
You know, the engine’s skeleton system essentially already does the no-stretching by design.
The skeleton is used as the “master” position when the bone settings are properly set to skeleton instead of Animation Scaled or anything else.
The engine then takes rotations into account instead of bone positions to prevent stretching.
If you focus on having accurate rotations you should be able to pull off the same effect of letting the engine handle whatever skeleton you feed into it just fine…
Calculating the rotations from the camera pose is where the load would be at. As such, good software becomes imperative… then again its SOCATHOA so the math isn’t all that complex, theres just a lot of it…
