Sounds like the way ISM updates the positions of its instances does not get reflected in the velocity buffer needed for temporal AA. Have a look at this: [UE 5.0] Ghosting Issues with Instanced Static Meshes and TSR/Temporal AA
You could look into how you are updating your instance transforms and whether that is designed to work with TSR.