Actors pending kill unexpectedly

I created a dice manager class which holds references to dice actors it spawns, these dice rarely but still on occasion end up being destroyed without a call to destroy them in my blueprints.

Would I be better off simply destroying the dice once I’m done with it and spawning a new one when I need to roll it?