If you need to swap actor ownership around, perhaps you do not need a CAC. You’re not taking advantage of what it does and it seems to be gettting in the way here.
Attach actor to actor instead - less fuss. And one fewer component to worry about.
CAC is great if you want to Set Class. The underlaying actor can also travel up the chain to find the CAC’s owner effortlessly.
Perhaps it’s the wrong tool for the job.