First off, it is complicated. I wouldn’t suggest trying to understand in too much detail how it all works, just in order to make use of it.
So, a few things.
Components get special treatment. There is a whole lot of code, in-game and in-editor, that is written for dealing specifically with components. For one thing, all components are Instanced by default, even if you don’t mark them as such. Further, there are special customizations for displaying components within the details panel. That enables you to edit properties of default components, but you can’t do the same for regular objects. Currently that can only be done if they are Instanced, which gives you the rather different looking inline expansion. I think this is something that could be improved, to support for all UObject properties many of the things that currently are supported only for components.
A note about Instanced. It’s a bit of a confusing name, in that even object properties that are not marked as such, will still point to an object instance (when not null). Essentially, the difference is in how they are treated within the editor/details panel, along with how they are treated for object duplication purposes - instanced properties will mean that a new copy of the object gets created whenever its parent is duplicated, whereas with regular properties, a duplicated parent object will just end up with a pointer to the same subobject instance as the object that it was duplicated from. So, Instanced results in deep copying.
The dropdown for an Instanced property gives you a list of classes, but this is because it’s saying ‘I’m about to make a new object instance, tell me which exact class you want me to make it’.
TSubclassOf is just a typesafe wrapper; essentially it’s the same as defining a property of type UClass*, it just limits the hierarchy of classes that can be assigned to it. As you say, it stores a class, as opposed to a regular object instance.
The one other thing that comes into this, and can get quite confusing, is assets. An asset is, for the most part, just an instance of a particular UObject type, that exists (and persists) at design time and is listed in the content browser. So a particle system you make is an object instance of type UParticleSystem. It does not represent a new type of class, hence it won’t show up in the list of a TSubclassOf< UParticleSystem >. The confusion comes with blueprints. A blueprint is actually an asset which is an object of type UBlueprint, but it represents a new class, of the same name as the blueprint. So blueprints (or rather the classes generated from them) will show up in the dropdown lists for TSubclassOf properties.
So finally, going back to regular, non-Instanced UObject properties - this dropdown is asking you to select an existing object instance. At design time, this is essentially (with the exception of actor classes) any assets you have of the given type. If there are no assets, then the list will be empty.
Hope that clears things up a little.