Collisions will also get you here (performance).
A lot of games use a sphere around the player to check which things are in ‘interactivity-range’ so you can only enable stuff you need to when it’s appropriate, otherwise most costly-things would be disabled or ‘turned-off’ in terms of physics, etc. Think how souls/etc would need to check if something is in range to lock-on to the critter, but with other stuffs.
In your case, however, I’ve been mulling it around and unsure what you can do. I was thinking you could keep a library of simple collision-shapes and muck around with those, but unsure how performant it might be… You wouldn’t want collision on anything unless it’s just-about to be aggregated into the ball. Do collision-checks from the ball to anything within range-X.
Collision-wise, overall, I would think you will always start with a sphere and it’s more what is sticking OUT from the ball-of-things that makes it roll weird (I did play this game). Taking a sphere-collision and then welding a capsule to it but such that it sticks out so it rolls oddly? Then as you add more and more, you can recheck the configuration of your ball and adjust accordingly. Maybe each mesh has a simple collision-type (sphere, cylinder, cube, etc), and just attach ‘that’?
Things of a certain size that are smaller vs the current size of the ball would add little/no collision so as it gets bigger, you could get away with testing less-and-less?
Musing out loud, hope it helps.