Just found an interpolation method that fix the jittering problem. It still remain a small laggy movement when orbiting around player character, but following is better.
If anyone interested:
http://forum.unity3d.com/threads/how-to-smooth-damp-towards-a-moving-target-without-causing-jitter-in-the-movement.130920/#post-884233
If someone has solutions you are welcome, I’m always searching for a better way.