I’ve been working on replicating the Deep Mimic paper within Unreal Engine and a few week back I started using Learning Agents. It’s been a massive help so far. Both simplified, and sped up the training.
My goal is to continue with more complex animations, beyond that more complex characters, and ultimately create games using this technology. Still a considerable distance from that goal though!
I haven’t delved too much into the code side of Learning Agents (just been culling a lot of the logging so far) so everything currently is achieved with Blueprints. In the future I’d like to create some functions to handle parameters such as Learning Rate Decay, find a way to have more control over number of neurons per layer, etc.
I will continue to post my progress in this thread, along with any queries regarding learning agents. Happy to answer questions too of course.
Trained a second agent over the weekend though was pushed for time so this hasn’t had as long to train as my baseline. Animation is slightly more complex.
(agent on the right)
Holds up ok though I feel with some tuning of the rewards and hyper parameters I can get this looking much smoother.
Away for a week now though so progress will have to wait until I return.
Since I am working towards a commercial product I have no intention of sharing source code at this time. I am happy to answer questions though and share my experiences developing this so far.
Back from a week off, so back to training. I developed a walk cycle a while ago which I will attempt to train next. I am not expecting it to learn this animation first time. I feel adding movement will require a level of tinkerage.
In the deep mimic paper they add reward for forward velocity. Currently I do not have any exta rewards outside of mimicking the animation. Will share progress once training is complete.
It’s been fantastic so far! Having followed your drive tutorial the blueprint API all made sense. Simple transitioning what I had set up originally.
Only issue I’ve encountered so far is with logging. I’d appeciate more control over what gets logged, and an option to disable logging altogether. I had to remove logging when agents restarted due to the volume of output that was being generated, and now that I’m using tensorboard ideally I’d like to disable logging entirely. I see in the .json file “logging” is a variable so it may just be I can’t find it.
Super excited to see what’s coming in the next update!
Just an update to say I’m still working on this! It’s been quite boring as far as visible progress following Christmas. Spent most of my time finding bugs and redoing the reward function. I’ve been training agents daily and there has been improvement which I will share in the coming days.
I’m starting to see why Deep Mimic never really took off in the same way Adversarial Motion Priors has, as tuning everything to get a good result is a big time commitment. Still, that paper is for another day!
Last couple of months have been irorning out the problems seen in my work so far. I haven’t wanted to leap ahead with human characters or challenging monsters before getting the basics right.
And here is the anim ref (created in blender)
If you watch the reference you’ll see my animation doesn’t loop properly. This is because I was a total wally and forgot to check my export. It explains why you see a slight pause in the animation during simulation. It threw the phase observation out of sync!
I wanted to confirm the stutter in the previous video was related ot the animation cycling so I trained it again this afternoon.
I really like how when you pull the Ant away from its intended direction it naturally steers to head back on course. Exciting to see stuff like this, makes the process feel worthwhile.
Finished making the needed changes to both my UE code and Bullet to allow for more complex characters to be included. Will be setting up the joints and collisions on my biped character today, and then discovering what struggle lies ahead in getting them to mimic an animation.
Been one problem after another setting up a biped in Bullet physics, and I feel it becoming clearer now why the original paper opted for their own PD controller to handle joints and motors. I’ve been using the spherical motor and constraint for a MultiBody and while it’s worked up until now, I’m finding a real lack of option to tweak the joints for them to provide the motor with more force. This is tricky when you have an arm hanging off of a weak body.
I’ve looked into how Bullet imports a URDF, and rather than using spherical joints it will combine hinge joints to allow for 6dof. While this could introduce gimbal I’m hoping it will at least allow me to get some simple animations up and running while I opt for a permenent solution. It’s been a while since I’ve looked at the physics engine in Unreal so maybe I could just do away with Bullet at this point?
I have thoughts on why it may be struggling. The first is that even a small amount of noise can have a drastic impact on the action taken, so will try again with noise reduced significantly.
The second, which is also mentioned in the paper, is that the character may just need further tweaking so that it is capable of this. I see in the original code they have their own PD controller, and while I experimented with my own PD controllers in the past I settled for the motors provided with Bullet. I’ll admit though, I’ve often had problems with these. Mostly erratic behavior and constraints being broken, enough to suggest they’re not as rugged as they may need to be.
Up until now most progress has been made after significant improvements to the environment rather than slight tweaks to hyperparameters, so I feel the next steps are to familiarise myself with the maths required to build these PD controllers and make such an improvement to the environment. I’ll continue to train different parameters, more so for data than anything, but may be a few weeks before I have any significant progress.
I can see there is a new version of Learning Agents coming soon but may have to hold off on that update for the time being (Unless it comes with Deep Mimic included in which case these past couple of years are now redundant! haha)
