Whoops, I marked it accidentally as solution, it does have a physics asset and still has this problem.