Perhaps this should be in the Player Controller. This way whatever you click on gets destroyed.
To filter things out (so you don’t destroy everything you touch!), instead of casting, you could add new object types in the Project Settings.
The above will detect only one specific object type which can be assigned per actor / class. Quite efficient.