Hi ShrewGlue,
Not sure this is the ideal way to manage this but you may want to try “Fracture impulse Scale” in your destructible mesh detail.
See attached
Edit: based on my own experience, do not hesitate to “push” the value. You need big scale to witness differences.