Without any actions, the performance is great in both cases, maybe even slightly better in UE5. (around 2-5 ms in my case)
The “Add/Remove” operation cost scales regardless of the amount of spawned actors (tested all that too).
Actually you are right, Collision seems to be the problem. Why is UE4 much faster in that regard though? 