Update, so after more investigation it is not AsyncTask, although still some weird behavior here, so perhaps I should use another solution for async logic.
The culprit of the slow down is the collision building when adding instances via HISM. Still, I have no idea why a packaged build would be significantly slower. Anyways, as this post is now misleading I am moving it to a better one.