Make >physics material< set damping force to 0.7 or even higher, this way it will not bounce back. May still randomly spin once or twice. Physics engines in games have that weird behaviors, nobody can solve really, they just hide it more or less.
Ps.
From watching video it may be colliding with part of player character. And since player character (or animated skeletal meshes) have infinite inertia (they affect physics but by default are not affected) you player character may push that yellow box with quite big forces.