Ok I apparently fixed the issue, it was coming from my EmptyProps which had a low poly editor only mesh (no collision, use simple as complex), but replacing those by a real empty prop worked.
Maybe this is because of distance fields I use in the ball material? Really strange though