We had that issue too and it turned out it was a problem with Windows not allowing Swarm Agent to create the Jobs folder inside the \DotNET\SwarmCache. So, we manually created the folder and that fixed it.
Looking at the date this question was posted I’m guessing you already worked it out but I thought to write it down in case someone else comes across the same problem.