Before yesterday I had decreased most of the texture sizes from around 4K to 1K, still had the issue, then yesterday I remembered the textures used for the sky.
It is actually made for the users who need dynamic cycle. My sky is static but I really like the cloud mixtures you can achieve by it. But most of them are useless while it BP uses all the textures. It has 5 4K textures for each day, night, overcast and sunny skies. I set all the unused textures of it to 256 px, didn’t lose quality since they are not used. Before then my game ran buggy on 3 GB VRAM, now it is stable even on 2 GB VRAM!