What's the API editor use to select a actor?

In order to get a reference of the selected objects you can use the GetSelectedActors function located here:

The next step in order to get a specific actor class from the selection above is to call the GetSelectedObjects functions inside USelection class:

I doubt this logic can be used by a shipped game, however you can use it inside an Editor plugin or maybe during your development.