On UE 5.2, I remember having some troubles with that function. Sometimes it works, sometimes not.
Example in a construction script: set the collision profile of a component that have BlockAll profile to NoCollision => this works. Later on, do the opposite: set back the component to BlockAll profile => nothing is changed (it remains in NoCollision profile)
Very strange. It is probably a bug, or something I misunderstood about that function.