So after some researching on my own, I found out that for this functionality a customized Player Controller is not needed. I can simply register all the bindings in the pawns like this for example:
void AAvatar::SetupPlayerInputComponent(UInputComponent* PlayerInputComponent)
{
Super::SetupPlayerInputComponent(PlayerInputComponent);
PlayerInputComponent->BindAxis("avatar_move_forward", this, &AAvatar::MoveForward);
PlayerInputComponent->BindAxis("avatar_move_right", this, &AAvatar::MoveRight);
}
If I then access the player controller of the pawn that is currently possessed, and make it possess a different Pawn all Inputs for any other Pawn will be ignored. This makes (in my case at least) any more complicated managing unnecessairy.