Probably you made this problem by using very complex object with huge number of UVW islands and overscaled LM resolution ( i.e 1024px+).
Sometimes is better to have complex object as separate sub-objects with own Ligthmaps and simpler UVW and then put them together into an single Actor Object.As I noticed the SWARM uses single CPU thread for certain lightmap. So even if you have 128 CPU threads in your computer and just a single one very complex LM then only single one CPU thread will be used for calculating that complex LM ( !!! insane waste of CPU power )
Very long CPU afterprocessing can be visible also when you are using very dense VLM cells ( very small size like 1cm ).