Merging animation and MIDI input animation

Development Character & Animation
, , , , ,
1

Hi,
I’m actually trying to make a videoclip (cinematic) where a character plays the piano.
Here is my plan :

Make a blueprint that manages midi info from my keyboard and moves the keys according to the pressed keys. This can be recorded in a DAW and played back later on (So that I don’t need to have UE opened during performance capture) (This is done in 4.27.2)

Capture the actual performance, I’m going with a visual AI app (This one seems to have better results than actual gloves/suits)

Merge the two animations. (In UE5)

Issue :
#1 : The piano mesh and keys has to be the exact size of my real piano.
#2 : The character needs to have the same finger size than me.
#3 : The fingers have to land perfectly on the key.

Question :
Is it gonna work ? Is it an actual workflow that makes sense or am I going in the wrong direction ?

I feel like I might be going the wrong way, but at the same time it looks like something that is being done a lot in the actual film animation industry.

Thanks a lot for your help :slight_smile:

Capture d’écran 2022-03-14 à 21.36.04
Capture d’écran 2022-03-14 à 21.36.041920×1088 154 KB

2

Update :
I’m having a second thought about the solution : If the finger animation is quicker than the actual note animation, the finger might go through the note mesh.
So one idea would be to activate collision of the note before moving it and deactivating it after release. It would add a problem if the finger goes through the mesh prior to the activation of the collision.
This would help with not working the animation by hand afterwards.

Any thoughts about this idea ?

3

Hi there !
Anyone having any idea ?
Thanks a lot :slight_smile:

4

This maybe late, but there is an unreal engine 5 plugin called MidiEngine.

You can use it with the Modular Piano Keys plugin to animate piano keys…