Agreed, watch out for cross referencing assets, this easily can make web of references all over your level. And everything referenced byt some always loaded actor will also be always loaded.
So player controller, referencing all custom door actors is really bad idea.
But all custom door actors referencing single blueprint is kind of ok.
However best is if you use soft references.