Sounds like what you want is to randomize the time input. The basic Idea should be to pan over a random noise over time and use that output as the time input to your panner.
Here is a quick example using two noise nodes with slightly different scale to unsync u and v panning.
