So I finally figured this out. Apparently, the reason why the physics is behaving weird is because of there is by default a sphere in the center of the world, which represent player start.
More about the sphere on this post:
Why is there a random sphere in the middle of my scene when rendering a video through Matinee?
Since my character just so happens to be in the center of the world, apprently the sphere is colliding with my characters. I do not have the “player start” in that level (since I am not making a game), I assume the sphere does not know where to go and so idles at the center by default.
With that being said, to fix this, just create a “player start” actor and move it away from the character.