It’s normal when building in higher quality levels.
Lightmass doesn’t split the load among the multiple workers evenly, so depending on your scene all other threads can be done with their work sitting waiting for the one unlucky thread who ended up with the largest model in the scene.
It doesn’t matter how many threads or render nodes you have, the time for building the lighting is limited to how long one thread takes to light the most complex model in your scene.