Hey weberl,
Have you taken a look at the Swarm Agent Troubleshooting Guide to see if there are possible solutions for your issue?
Have you attempted to build the same project on a different machine not connected as a build farm?
Try increasing the verbosity of your logs to see if there is any other information to be obtained from your issue.
Thank you,