Thanks Eric. I started removing objects one by one and found the culprit. Nothing seems visually wrong with the mesh but a “Map Check” throws a warning saying it has “7 elements with 0 triangles”, and if I remove that StaticMesh then everything works fine.
Might be nice if Swarm/Lightmass was a bit more upfront with what the problem is, but I guess this solves it for me. Or at least points me to what I need to look at.