Swarm - What is it and how to use it?

Development Platform & Builds
, , ,
1

So from usage and Google searches re UDK I am guessing that Swarm is a program within the Unreal4 engine that manages mulitple clients to undertake distributed compilation of primarily Lightmass solutions but possible other aspects of maps.

My questions for you fine folks are:

  1. Am I correct above in describing Swarm or does it do something differently than this?

  2. Is there documentation available for its set up or operation?

  3. Do I need to install the full Unreal Engine on the distributed clients or is there a sub-set I can use (what is the minimum client to place on slaves?)?

  4. [Unlikely I know but…] Is there Linux version of the client? If not, has anyone had success using a windows VM under Linux?

  5. To start the client do I just run the “SwarmAgent.exe” found on my Windows Box at C:\Program Files\Unreal Engine\4.0\Engine\Binaries\DotNET ?

1 Like
2

  1. That is correct, it is used to distribute lightmass. (And nothing else)

  2. If documentation is still limited its operation and interface seem to be identical to the ue3 version so you could also look at documentation written for ue3/udk.

  3. You probably won’t need everything, although I’m not sure which part that would be.

  4. No, and no idea.

  5. That is correct, although you also need one coordinator running on your network to manage the different agents.

3

So from usage and Google searches re
UDK I am guessing that Swarm is a
program within the Unreal4 engine that
manages mulitple clients to undertake
distributed compilation of primarily
Lightmass solutions but possible other
aspects of maps. {…} 1) Am I correct
above in describing Swarm or does it
do something differently than this?

Yes, Swarm (short for UnrealSwarm) is a standalone C# application that distributes units of work across multiple machines on the network. Currently, Swarm is only used in conjunction with Lightmass to pre-compute lighting for levels, but the application itself is task agnostic, which means that it can work with any other type of task as well. We just haven’t taken advantage of that yet.

  1. Is there documentation available
    for its set up or operation?

UnrealSwarm was created for Unreal Engine 3, and it looks like the documentation hasn’t made it into the UE4 Docs web site yet. That being said, Swarm hasn’t really changed (except for a few bug fixes), so the old documentation still applies. You can find it here:

http://udn.epicgames.com/Three/Swarm.html

http://udn.epicgames.com/Three/Devel…#Swarm%20Setup

  1. Do I need to install the full
    Unreal Engine on the distributed
    clients or is there a sub-set I can
    use (what is the minimum client to
    place on slaves?)?

No, you do not need to install Unreal Engine on the Swarm agent machines. Swarm will automatically transfer all files (including binaries) that are needed to process a given job. The easiest way is to copy the /Engine/Binaries/DotNET/ directory from an existing UE4 installation to a new computer. Then follow the instructions on the pages linked above.

  1. [Unlikely I know but…] Is there
    Linux version of the client? If not,
    has anyone had success using a windows
    VM under Linux?

UnrealSwarm currently only runs on Windows operating systems. You may be able to get it to run in a Windows emulator on Linux. We briefly had a version that worked on MacOS, but newer versions of Mono introduced some breaking changes that we haven’t fixed yet. We are planning to rewrite UnrealSwarm in C++, and at that point it will also become cross-platform and seamlessly work on Linux and MacOS. There is no timeline for this project yet, but it will probably happen this year - possibly as early as this Summer.

  1. To start the client do I just run
    the “SwarmAgent.exe” found on my
    Windows Box at C:\Program Files\Unreal
    Engine\4.0\Engine\Binaries\DotNET ?

Yes.

I can’t answer #5 yet because when I
execute the executable I get a
“Register for Visual Studio C++ dialog
box” - Made me laugh that an
executable needs registration with
Microsfot to run, it has been a long
time since using Windows

You may be missing some prerequisites for UnrealSwarm. Make sure that the latest version of the .NET Framework is installed on your computer. You may also need the Visual Studio redistributable package. The easiest way to install prerequisites is to install /Engine/Extras/Redist/en-us/UE4PrereqSetup.exe

4

How much data is actually transferred at the startup of swarm to the slaves ?
Depending on this, it might or might not be interesting to offload lightmass cooking to another site (assume a vpn connection), or even a cloud.
No sure if it is possible, but Azure springs to mind, and if the lightmass agent and swarm is ever improved to a linux version, any cloud would theoretically be able to offload this heavy process.

5

The UnrealSwarm usage itself is rather neglectible. The problem would be with Lightmass data. It highly depends on the level, but it may rise up to several Gb for large levels.

You can find out yourself how much data is there looking into the Swarm job folder in swarm cache for your concrete level.

So it looks like there is currently no way of transfering this data through the Internet (depends on bandwidth of course :)). At least it wasn’t designed for that.

6

I’m assuming the swarm computers do not require a GPU?

7

Also, how much ram would a swarm pc require?

Assuming 4gb will be fine since it doesn’t require the Unreal Engine to be running?

8

Hi ,
this is more Lightmass question rather than Swarm, cause Swarm only copies and deploys Lightmass tasks to remote machines.

Lightmass does its computation on CPU only. Its highly multithreaded, so the more cores the better. It is also recommended to have 1 Gb of RAM per thread (at least for big scenes). You’ll need to have WinXP or higher. There is also a strong recommendation for 64-bit system as it’s 30% faster and you won’t hit the out-of-memory exception if you install on that machine more than 4 Gb.

Sorry for the long response, but we have X-mass break :).

Jarek

9

Has this changed since Apr 2014?

Specifically: is SwarmAgent available on mac now and is it possible to distribute the lightmass calculations from mac to remote PCs?

10

@devel.bmad strange isn’t it, many many months later and it seems no one else wants to distrubute lighting rendering.