Ambrosiussen, I missed the part about you not baking at High or Production quality! I think you may have to temper your expectations about being able to get fast bake times and perfect quality. Fast bake times (for quick iterations) require quality tradeoffs.
If you want to speed up bake times for High or Production bakes, I’d recommend setting up other computers on your network to act as Swarm clients. This will allow the processing to be distributed to multiple computers and thus speed up your lighting builds. You’ll find information about how to set up Swarm here: Swarm - What is it and how to use it? - Platform & Builds - Unreal Engine Forums