I’m completely new to unreal engine, but my task is to teach (preferably two) AI agents to dance together (or do something resembling it) using UE’s built-in physics engine, but I have no idea where to start. I’ve seen some plugins that may be useful for the task (MindMaker, ONNX, Learning Agents), but didn’t really find sufficient information to get started with anything from the bottom. Do I even need such plugins to make this project or is it possible with baseline UE? Could someone point me in the right direction?
I think this is very possible, but will require lots of tinkering before the training can properly start. Personally, I just started using Learning Agents, and wrote a quick tutorial here. It’s become a bit outdated already, but at least it can help with installation and getting started.
Aside from the machine learning, I think one of the bigger challenges here is to control the limbs of a character. For this I recommend checking out control rigs, and getting familiar with those, and then have the AI system interface towards that.
Assuming you have a training setup ready, there is still the challenge of teaching the agent what dancing is, and what it isn’t. Initially, you could download a bunch of dance animations from mixamo, and try to imitate those in some way.
Also, google did a similar project once, so you could check out that for inspiration: AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
I hope this helps you get started! Good luck, and feel free to reach out! 
1 Like