Strange distortion of particle system caused by motion blur

When moving around - everything is fine.
But when rotating around objects, my candle flame particle systems start to spaz out, unless motion blur is disabled.
Here’s what I mean:

That doesn’t look right, especially considering that the flame is constantly staying at the center of the screen.

Is there a way to fix that, besides disabling motion blur completely?