It took a lot of experimentation, but I figured it out.
The solution is to change the runtime hash class to WorldPartitionRuntimeSpatialHash instead of the default WorldPartitionRuntimeHashSet.
The difference between these two is not documented anywhere as far as I can tell, but this change made world partition behave the way I want for me.
And I only needed a single HLOD Layer.
The Loading Range in the HLOD layer then controls at which distance are the HLODs loaded in.
Loading Range in World settings controls at which distance are the actual actors loaded in, replacing the HLOD.