I want to generate skeletal meshes and animations based on existing video material (think running animals). Recent developments:
Is anyone successfully using these or other software in a way to create assets for unreal? A plugin for Blender would be epic.
AcinoSet A 3D Pose Estimation Dataset.pdf (8.0 MB)
ZooBuilder 2D and 3D Pose Estimation for Quadrupeds Using synthetic data.pdf (6.2 MB)
Creatures great and SMAL Recovering the shape and motion of animals from video.pdf (4.4 MB)
BKinD-3D Self-Supervised 3D Keypoint Discovery from Multi-View Videos (2.1 MB)
NatureProtocol2019 - DeepLabCut for 3D markerless pose.pdf (1.2 MB)
I’ve finished other parts of the project, now it’s time to look at this again.
I’ve been using DeepLabCut to generate 2D markers on animal bones from just videos.
Challenge is that it doesn’t actually seem to do any 3D skeletal pose estimation unless you set up multiple cameras to triangulate bone positions. Now that is 2 steps forward and 5 steps back for motion capture.
What I am hoping for is a program that knows a skeleton itself provides a lot of the neccessary information to turn a 2D video from a single camera into a 3D pose. Skeletal structure, bone angle constraints, bone lengths etc. But what program does this? What can I feed a video of a mouse to extract estimated 3D skeletal poses from?
Say I have the skeleton. every bone has a transform, length, and it’s known that twisting one beyond a certain angle is not healthy. Part of the skeleton is not visible on the single cam video because it is behind the body or behind an object. But so what? You can fill in the blanks on that data. Am I missing something? The skeletal structure and bone characteristics are known. Blanks can be filled with inverse kinematics or directly from the training data.
Just what program does it? 