The manager was introduced in 4.17 and was supposed to speed up compile time for larger projects. And it does in some instances - it works well for projects created in 4.17.
I’ve seen many people report compiling issues precisely like yours (including myself) when transitioning from previous versions, though. It some cases the compile time increased by an order of magnitude…
Side effects are that the compile time may not be as efficient. Admittedly, I’ve yet to run into a tangible comparison. The worst case scenario is that you will not see compile time improvements when moving to 4.17.