So it looks like I misread a bit of it, and misunderstood a bit more. The truth of it is all mostly laid out in this doc:
It makes note that unreal engine containers TMap, TSet, TArray, etc all behave in the way mentioned in the original question. The object pointers in them of course can go null via destruction, so storing nullptrs and managing the memory that the object pointers take up in the containers is up to the user, but the objects themselves are handled by the engine as they no longer have references. At least that is how I am understanding it.
My old rule of thumb was if it has anything to do with UObjects, its best to keep it as a UPROPERTY, but going ahead I feel like I can make much better decisions on this now.
This seems like a good enough answer for me, if there is something I missed or needs more clarity, please clarify, either way, thanks!