The final answer in case anyone reads this.
After rewriting and cleaning my code, the UI was more responsive but i still got a huge fps drop behind.
After a lot of tests, i found that the fps drop was only occuring when using text style, and had something to do with the font size.
When you define a text slate widget style in the editor, there is no default font, you have “none”, “none”, and no number for the font size.
I was only modifying the font size but let the font to the default “none”. This was what was causing the huge drop in performances.
When i selected a font (roboto, as it was the only one available), everything went 60+ fps, even with full UI deployed.
Moral of the story: don’t let everything at their default values 
Here, the absence of a default font was the troublemaker.
Man, so much time lost on that one !!!