Exactly. scaling the time node is like multiplying the current time by some value. since time probably started a long time ago, it will have the effect of going backward or forward in time dramatically.
ie if current time is 100 seconds and you change the speed multiplier from 1 to 0.5, you will instantly go from 100 to 50 seconds, appearing to blend backwards in time. The best way is to drive the value manually by incrementing a float variable. If you increment that manually by the current speed it will always be relative to now instead of the start of the game.