No you got the question right.
I got confused because of this weird issue I was talking about before. The debug output of all the game play tags was displaying all the tags the owner ever had, including those who were not supposed to be active.
Your answer is correct too. I had to set the duration of the effects to Infinite to undo the buff when the gameplay effect is remoed 