When you are optimizing the game, there are tools available that allow you to see how much memory you are using. Since your game is unique, there is no way to automatically optimize for your game, so you will have to manually create presets. This ussually corresponds to something like High, Med and Low. You have to choose what those targets are. So Low maybe 1GB of memory, Med maybe 2GB and High 4GB or more.
But you also have to figure out how much performance a card with on 1GB of memory has, as well as 2GB and 4GB, so AAA titles ussually allow minute tweaks of settings in their Graphics options. You have to choose what level of exposure you want to give to each setting.
So, one strategy is to create multiple sets of Textures, or texture levels that can be changed at runtime. But all of this has to be done by the developer. And the best way to know if your game runs on a system is to just test it. This is where alpha and beta testing comes in, it is more than just trying to attract attention, it is absolutely necessary for your games success.