Hi Marc,
The purpose of the Swarm Cache folder is to cache unmodified files that are required for building lighting, so that the Editor does not have to export them before every single build. Each asset that is required for Lightmass has a unique GUID that will remain the same until that asset is modified by you or someone else.
You are free to delete the contents of the Swarm Cache folder, but your next lighting build will then take significantly longer, because all required files will have to be exported again. If you are concerned about the size of the Swarm Cache folder, you can configure the maximum desired size in the Swarm settings.