GAS: Predictive Infinite GE lingers on client

I have been working adding movement speed GE that can control player movement during ability montage. The montage will fire events to apply infinite durationGE predictively, and then those GEs will be removed later after the ability ends. The issue is that if I have multiple events like this in a short montage, there is some chance after the ability ends, the client would still have 1 or 2 instances of the same GE lingering around forever, causing movement slowdown on player. If I apply those GE server only, the problem fixes itself. After some digging, I also noticed those lingering GE are replicated GE from the server, not local predicted GE.

Any help is appreciated