yes, i have also tried myself and you can not completely avoid pushing the vehicle and you can not be pushed by the vehicle in any way.
i read in a forum post that you can attach a kinematic collider to the vehicle that only collides with the character or attach a collider to the character that only collides with the vehicles to prevent physical interaction but not sure if it will work.