Hi Joe,
After profiling this intensely I found that the issues was just my perception of the graph. I was testing it at an unlocked framerate and thus the difference was visible. After capping it at 60fps there is almost no visible impact at all.
Sorry for the undue trouble!