Hmm, are you sure that your units are right then? For example forces need to be applied as Ncm instead of Nm.
The other potential issue - taking into account that you apply forces yourself, you can get similar bug when force is applied to an object which already collides with something. Like for example rocket engine propels rocket into a wall, if rocket hits the wall and engine keeps pushing it, it can get propelled into infinity.