I feel stupid, but I have to admit that I started panicking when I saw this thread and when I updated my android project to 5.4.
It indeed became more sluggish. But it turned out to be only for non-shipping builds: Wait For Task taking too long in Development Build