How can I make a destructible mesh collide smoothly before destruction?

I am currently working on a ‘roll a ball’ game, and would like to make a destructible sphere mesh roll normally on the surface, until it hits something with significant impact (in which case it gets destroyed). With a simple mesh it works fine, but once I use a destructible one, its rolling gets bouncy. Here is a gif of what I mean. The sphere behind is a destructible mesh, while the other is a simple static one with the same parameters. Ideally, I would like them both to behave identically.

I suspect that the problem is with collisions - destructible sphere’s mesh is made of meshes of its chunks, and their collision meshes aren’t as precise as of a simple sphere, as I understand. Perhaps I missed something, and there is some option to make those meshes work as desired. If not though, how can I make it so that the sphere would roll smoothly, yet would get destroyed once it hits something with significant impact?