GC extremely huge hitches in empty map

In standalone it’s the same, but in a cooked game the spikes are even smaller. But how can it be that adding more assets to the game does make the GC slower? Why does the GC check if references are invalid which never can become invalid?
I doubt that with those spikes it is possible for me to make a playable VR sports game. What do you think will the player do if he is just about to beat a level, then the game pauses for one second, he loses the focus and loses the level… I would throw the game in the corner and never touch it again. Maybe the best option is to turn the GC completely off and manage the memory by myself or take all actors and UObjects from object pools and never destroy them.