OK, in the end I solved it like this:
As foodi suggested, I update the target offset in the direction of the point that is under the cursor.
When I then start rotating the camera I interpolate 3 things:
- target offset back to (0,0,0)
- spring arm length so that the new size is equal to the length of the vector pointing from the camera position to the spring arm position (i.e. the center of the rotation)
- angle of the spring arm in such a way that the direction is the same as it was when the camera was shifted by the target offset
When the interpolation speed is not too high it makes a nice transition effect.