As I typically do, I ran a quick test to see if I could just reproduce the Swarm Failed to Kick Off issue using your description for the level setup, but everything checked out and built fine.
Use the suggestions on this page to go through and try to see if that resolves your issue.