cubemaps are kinda “expensive”. i remember back then they used 3 memory samples to fetch a cube pixel. add a random value like a normal map vector and it will add a coordinate dependency. not as streamlined as you think.
i mean… gpus are fast now, still a bit of trouble on the metal deep down. trash of cache. it could add up.