If you are on 5.5 I believe it had a memory leak if you were mixing sample types. Meaning you have like 64 temporal and 4 spatial. If you want motion blur put them all in Temporal and just 1 spatial. The opposite if you do not want motion blur. If you are in 5.6 then I am not sure. What does your memory consumption look like over time?