Some update here : I managed to get 5-fingers tracking working on my custom skeletal mesh.
However for some reason the Pawn BP can’t get acces to the Anim Blueprint of the mesh.
Casting to another Anim BP (set with another skeletal mesh) works fine.