When you exceed your RAM limit it should start to use your Virtual Memory unless if that is configured incorrectly. Perhaps you have a performance problem with your hardware RAM during heavy loads.
If the hardware RAM is the problem then you either need to reconfigure it or replace them but the problem could easily start somewhere else like in a poor Power Supply or Motherboard or the Disk that is being used for Virtual RAM.