Right. The idea I had was moving both primary tick and cloth tick to during physics only if you are not running end physics (in which case I leave everything as is). Based on what you said I should be able to do this because I am not simulating a ragdoll and also because the cloth sim only does local collisions since it runs in its own solver independent of the global phys sim one.
Is that correct?