panner is simply this:
Position + (SpeedVector * Time)
If you have your own “Time” value that you accumulate somewhere, you can simply write that formula and it will work.
The reason you cannot simply change the speed of a panner by scaling time is that it scales it from time=0, so in fact by scaling time you travel back in time. To scale time you need to instead add a new delta to your stored time variable.