Restraining character using Physics

I am trying to control my character using physics constraints and manual contolls and Id like him to maintain some default animated pose .

Here is my problem , if I simulate all all bodies he just falls , If I simulate bodies below

and for example connect his arm to the wall using contraint component he will just ignore it and walk away without his hand

so is there a way for me to make such constraint to affect main body without making them all simulate ?

Thanks

Screenshot_2.jpg