You should be able to filter out objects to which the field is applied.
You have to add a Field System Meta Data Filter to the actor. Then, I guess set the Object Type to Destruction like this:
