The physics asset is also just a bunch of collision boxes (or maybe capsules). How accurate it is depends on whether it is created automatically by the engine when you import your skeletal mesh, or how careful the tech artist (or whoever) that created the physics asset wanted it to be.
You can switch off your capsule collision just by selecting the capsule component in your character and tinkering in the Collision > Collision presets section of the details panel. And similarly you can change the collision response of the character mesh using the collision presets. So by default the capsule is set to the Pawn preset, which blocks everything except the ray casts on the visibility channel. The character mesh has the CharacterMesh preset by default, which blocks everything except ray casts, vehicles and pawns.
So to get the effect you want, you could just change the capsule to NoCollision and the character mesh to BlockAll but this is going to have a big impact on your gameplay. The capsule is pretty fundamental. I would suggest you proceed with a lot of care. I don’t know what use case you have in mind but it would be far safer to use custom collision channels just to cover that use case and leave the default behaviour for all the others. If you don’t believe me, then create a third person template with starter content and try to get the mannequin to sit in (or stand on, if you don’t have a sitting animation) the orange chair (SM_Chair).
For example, in a shooter game, you might create a custom collision channel for projectiles or weapons to stop some weird behaviour. So you might not want your projectiles to be able to collide with each other or you might want to make sure that a big weapon, that sticks outside the character capsule, to stop clipping through walls, doorways or the ground. You can create your own collision channel in Project Settings > Engine - Collision > Object channels.