The fractured geometry collection simply explodes at the slightest touch. It looks like a bug, but unfortunately my project is on 5.7. Maybe there is a better solution in 5.6, but if you animate the movement of the ball around the object, everything explodes anyway, but not so much.
5.7
5.6