AnimGen allows users to create character controllers using Machine Learning in Unreal Engine. This tutorial series covers the basics of AnimGen including how to train your own system from your own data, how to combine AnimGen with the Game Animation Sample Project, and how to combine AnimGen with MetaHuman Animator Markerless Motion Capture.
https://dev.epicgames.com/community/learning/tutorials/eGME/unreal-engine-animgen
5 Likes
@anorangeduck Thank you so much for making this tutorial series! I’ve been curious about about AnimGen since it first showed up in UE5 Main GitHub. The tutorials have been extremely helpful!
I hope at some point you make an additional video of the AnimGen Controller Behavior options. You go over Trajectory Follow Behavior thoroughly, just a quick run down of the following would be helpful!
- Encoded
- Idle
- Move to Target
- Optional
- Tagged
- Trajectory Interaction
- Uncontrolled
- Velocity Follow
In your paper/video of Control Operators for Interactive Character Animation looks like you add a lot of fun options, such as interaction with objects or other characters. I take it that’s what those Behaviors are for. It would be great on how to set them up and when to use different ones. All the Best!
Thanks @tomhalpin8 - that’s a great idea I will try to add another tutorial covering how to customize behaviors and how the tools relate to the Control Operators paper.
3 Likes
Thanks so much for creating this series.
I’m trying to get this to work with a charcter other than UEFN/Mannequin with about 6 minutes of data. Everything appears to be correct but I can’t get the character to animate via the trained controller.
the Database extracts rootmotion correctly. I can see the trajectories in debug
the autoencoder train correctly, I can see both skeletons, light and dark blue, matching closely, and trajectories same as animGen example
the controller trains correctly, I can see the trajectories in same way as the animGen example
All the blueprint stuff checked over and all match, works when I test it using UEFN sk and animgen controller
In play mode I can see the debug trajectory but character stays at starting point and does not animate.
*******************************
So it seems to be something in the data. I’m going through adding to the animdb anim by anim to see where the training breaks. any suggestions on making this go a little quicker?
I’d recommend checking the log for any errors. If the AnimGenController is outputting a T-Posing it means something is wrong with the input. Perhaps you forgot to add the “MakeControlObject” node in the EventGraph? Perhaps something misconfigured with mover if your character is at the origin?
Most likely issue with the data is something like not having the EnableRootMotion and ForceRootLock settings enabled.
Hello, thank you so much for creating these videos! They are very helpful. Would you consider making a tutorial covering the plugin more advanced capabilities? Something that really showcases what makes this plugin stand out, while also demonstrating how to use it in a more advanced and in-depth way. And also showcasing what this plugin allows us to do that other similar plugins simply cant? it would be really helpfull to see its unique capabilities and how to take full advantage of them in most advanced workflows.
@anorangeduck Thanks for the videos!
Quick question on scope. Could I train AnimGen on idle-type activity rather than locomotion? Things like chores, cooking, working at a bench, and then assign those as different styles for an NPC the way the sample controller handles walk styles.
Is that something the system is meant to handle, or is it more built around trajectories and movement?
Hi Daniel,
Is it on the roadmap to use a target pose as conditioning? This would be an ideal additional feature for our project.
I’ve seen a some auto-regressive networks be able to take a conditioning input, where the target pose influence fades up coming into the keyframe, and is basically injected into the prediction with increasing strength as you approach a keyframe. This would allow pose-to-pose animation, such as sitting down and getting up, where the sit down is a sequence of keyframes, i.e. stand next to chair → in front of chair → sitting → leaning.
Admittedly, these are diffusion models, such as GitHub - Yi-Shi94/AMDM: Interactive Character Control with Auto-Regressive Motion Diffusion Models · GitHub , but they’re still auto-regressive, and don’t actually diffuse a whole window.
I’m just wondering if there’s a way to produce similar behavior with your VAE model.
Thank you for making this. I gave up on integrating some of these models into Unreal, and settled for motion matching, but I’m going to use AnimGen now that it exists.
Per
Hi @Pieman468 - it is possible to train AnimGen on idle-like activities however it definitely isn’t the system’s strength.
@anonymous_user_7b0094b91 - I did some experiments conditioning on future pose. It is technically possible in AnimGen if you make a custom Behavior however I didn’t get as good results as systems which are dedicated to doing that kind of Motion Inbetweening task - so hopefully we can offer some more dedicated tools for that in the future.
1 Like
Hi Daniel,
Is there a proper forum for AnimGen, or is this thread it for now?
Everything is actually great, and very mature for a first release, except the training method. Since it’s hard-wired to train from inside the Editor (including using shared RAM with the Python script), it’s impossible to intercept. I now spend my days looking at progress bars and YouTube, and it’s a real problem. I’ve looked into various virtual desktops with GPU or upgrading another machine, but every solution brings new problems.
It wouldn’t be that hard for you to support all cloud GPU services like Vast.AI or Trooper.AI, and then people could train at 1,000% of their desktop speed costing maybe $5 per training run, while their computer is free to work.
Basically, the UI would have an option for Cloud training. There would be a field for SSH connection string, and perhaps which SSH library to use. The user then then starts their cloud GPU, which you have nothing to do with.
With an SSH connection in hand, you then install the correct versions of Python and CUDA, upload the corpus via a file, upload the training script, and start it running. The script would write status and snapshots to files instead of memory, and with SSH staying connected, you check those files every minute, and update the same progress bars. You also download snapshots every 1,000 iterations, and you can show them in the UI the same way. When the training is done, you tell people to turn off their cloud GPU.
Issues that will come up:
- If people don’t use an SSH agent, you’ll have to deal with them entering a password, making this an interactive session. I would therefore require that an SSH agent or 1Password is used, so that authentication happens entirely outside of Unreal Engine. Then you don’t need a UI for it.
- I would steer away from any attempt to help manage the infrastructure. You should know nothing about where you’re running. It’s the user’s responsibility to shut down the cloud GPU.
I think that without improving this, AnimGen will see significantly less usage, because people who aren’t experienced with the tricky personalities of VAEs and auto-regression can never know if their input is wrong or the model isn’t trained enough, and iterating costs 6 hours of productivity for every stab in the dark. They will judge the result after half an hour of desktop training, and conclude that the whole thing doesn’t work, and they will give up. I myself am trying to figure out how to completely rearrange my life in order to work around AnimGen.
With cloud GPUs, people can run full quality training runs in half an hour, while still being productive. I think it will dramatically affect adoption.
Per
Hi Daniel,
Also, some much needed feature requests.
- The base classes for the various attributes like AnimDatabaseFrameAttributeFunction and AnimGenBehavior need to become public ASAP. It’s impossible to do any attributes in C++, so you have to basically make a Blueprint shim in order to use them at all, because you can only subclass them in Blueprint.
- Same for AnimDatabaseFunctions. I again have to write them in C++, and then do a totally needless parameter-for-parameter wiring through Blueprint because subclassing these important classes is forbidden in C++. Is there a specific reason for making even the base classes private?
- I did develop one function that you may consider, Detect Gait Frequency. I’ll post it to GitHub if you’re interested. Basically, the network sees all gaits as equal, and it learns from the majority material that “a gait is a cycle of one second”. So slow movement just results in a lot of small steps that are still 1 Hz, not slower walking. The network needs to be trained to compartmentalize speeds separately.
Per
Hi PerHolmes,
Thanks for the feedback.
Supporting cloud training is definitely something we want, but as I think you realize, it is not at all simple. We can’t tie training to a particularly third party, and if we have to handle any user data it must be done extremely carefully. Until we come up with a plan my suggestion is to try and get a second machine or cloud machine which can run the unreal editor and do training there while you continue to work on your first machine.
Apologies for not exposing those classes as public - that was an oversight on my part. If it helps, these are now exposed as public on Main.
Hey! Im being testing animgen and looks awesome, thanks for the tutorials they are very well explained. I see the implementation of server training is a bit complex but I want to know if is possible to implement checkpoints or some type of savings during the training to stop and retake later. The distilling process takes a lot of time and will be very useful if that could be implemented in some point 
Hi @anorangeduck,
that’s awesome, thank you so much for bringing this to UE (I was already a fan of the 2020 “Learned Motion Matching” paper / presentation).
I have a question: for the different styles, does it need a full dance-card motion library for each style?
Would it be possible to do a delta between two styles and then use this? There are some papers which successfully did that - where only a handful of animations with the new style are added and it learns what the delta movement is and applies it to the full motion-set (e.g. a styled walk-cycle and styled idle might be enough to add the style to previously generic turn-starts and turn-stops).
Do you foresee this to be coming in the future?
In any case. Super exciting and I can’t wait to get my hands on it.
Thanks! 
I’m having an issue with the Animgen+GASP part of the tutorial not training the dataset. I’m very new to this kind of thing, but for some reason after following your exact steps it just shows training not started.