Distributed HLOD build leaves untracked files in __ExternalActors__ (Perforce cleanup needed)

Development Pipeline & Plugins
, ,
1

Hi everyone,

We’re running into an issue when generating HLODs in Unreal 5 using distributed builds across multiple machines. After running the builders, we’re seeing leftover files in the __ExternalActors__ folder that Perforce marks as untracked. To clean things up we currently need to run a p4 clean. Is this expected behavior, or are you running into the same issue?

Issue

  • After each builder run, new files remain in Content/__ExternalActors__/<LevelPath> as untracked changes.
  • Cleaning requires p4 clean, which is not ideal.
  • It would be expected to have these files automatically added to a Perforce changelist, since we could revert them quickly.

Setup

  • Unreal Engine: UE5.x (World Partition + HLODs)
  • Source Control: Perforce
  • Build: 24 builder tasks dispatched on a pool of 12 machines (so each machine runs two tasks)

Setup command:

<PROJECTNAME> <MAPNAME> -run=WorldPartitionBuilderCommandlet -builder=WorldPartitionHLODsBuilder -DistributedBuild -SetupHLODs -BuilderCount=24Builder command (per machine):

<PROJECTNAME> <MAPNAME> -run=WorldPartitionBuilderCommandlet -builder=WorldPartitionHLODsBuilder -DistributedBuild -BuildHLODs -BuilderIdx=<IDX> -AllowCommandletRendering -unattendedFinalize step:

  • Once all builders are done, we run finalize and then submit

What each builder does

  1. p4 sync
  2. Copy the HLODTemp folder from a shared drive
  3. p4 clean (will catch files only if the machine ran previously)
  4. Run HLOD generation
  5. Copy results back to NAS

Repro Step

On your local machine:

  1. Run setup commandlet
  2. Run builder commandlet
  3. Reconcile the __ExternalActors__ folder in Perforce → see untracked files

Any insight would be super helpful. Let us know if you need additional information.

[Attachment Removed]

2

Hi Hugo!

This is most certainly an issue we never ran into because our Horde jobs are always starting from a fresh workspace…

Can you tell me what are the files that are left over ?

I see that UWorldPartitionHLODsBuilder::CopyFilesFromWorkingDir() should be doing SCC operations (if enabled)… I guess the modified files are coming from somewhere else…

Thanks,

Sebastien

[Attachment Removed]

3

Hi Sebastien,

While running some additional test, it seems our build machines were using a bad set of command line arguments. We’ll be running some test this week, but I am confident this problem will be unlikely to rehappen again. I’ll get back to you once we have more informations.

Thanks,

Hugo

[Attachment Removed]

4

Hi Hugo! I’ll close this ticket for tracking purpose. Let us know if you run into issues again.

Regards,

Sebastien

[Attachment Removed]