Maybe have them always load, but set up the Cull Distance for them so they only render when close to the camera?