Real world implementation detail of "Build-time Asset and Plugin Exclusion"

Development Pipeline & Plugins
, ,
1

The article and workflows provided by Build-time Asset and Plugin Exclusion | Tutorial are really useful (to the point something like this probably should be rolled into the codebase natively for anyone that needs a game with regular feature updates).

There was one question that came up when discussing this internally, the example overloads PostLoad and PreSave with fairly boilerplate code that is most likely going to be the same for all classes. In actuality in Fortnite’s codebase, is that what happens? This is duplicated around? Or is it just presented like this in the example code for simplification reasons. The same goes for UExampleDataAsset::GetAssetRegistryTags.

I can see there being a world where any actor and any data asset in a game project that needs to support version functionality have a common base class? If this was a later addition to Fortnite I understand this may not have been feasible there, but just figured I’d check in case there’s something we didn’t consider.

[Attachment Removed]

2

Hello! I’m the author of that article and I’m happy to hear that you’ve found it useful.

Regarding your questions following from that article:

  • The PreSave() implementation to double-check that something is not unintentionally packaged by being referenced, we have duplicate logic in a couple of classes, but it’s only a handful. As you guessed, things like item definitions, quest definitions have shared base classes that implement it. If you want to implement a check like that in one place, you could perhaps get it to work with “export validators”. In [this [Content removed] I shared a snippet, jump to the snippet with “AddExternalExportValidation”. I believe those also run at cook-time, so you could do cook-specific save validation and use reflection, for example TPropertyValueIterator (snippet), to find your release version property. Or look for the object implement an interface.
    • One reason we only have the PreSave check in a handful of places is because most seasonal content we put in a GameFeaturePlugin, which in its entirety is included or excluded. It’s a bit safer and simpler to track than granular assets. We use the AssetReferenceRestrictions plugin to ensure there are no unintentional references from assets between certain plugins, i.e. avoiding circular dependencies between GFPs and avoiding main project /Game/ content referencing GFP assets directly. Since the GFPs are explicitly enabled or disabled via release version themselves, even if something would references assets in them, you’d find out at cook-time.
  • For ::GetAssetRegistryTags() it’s the same answer as PreSave: implemented duplicatedly in that same handful of classes.
  • The PostLoad() solution of excluding an actor from a map at cook-time isn’t something we use in Fortnite. It was a trick that I found interesting and wanted to share, but don’t consider it as battle-tested or “Epic does it”. For optional map content, you can use External Data Layers though! That feature came out after I published the article. In fact, I’ll add a note in the article to point people towards that.

Now to add a little of my own opinion: I would probably try implement a solution so that you don’t have to implement the same GetAssetRegistryTags() and PreSave() function for assets that can be granularly excluded. The FSavePackageSettings::GetDefaultSettings().AddExternalExportValidation snippet in that linked thread is something I would explore. Any UObject that would have the version range interface, or struct as a UPROPERTY, can get its version checked there. I haven’t tested the idea, but you might find it interesting to try out.

If you have any follow-up questions, do let me know!

[Attachment Removed]

3

The solution to discover plugins in the Target.cs logic is intended to work with RunUAT BuildCookRun, and other scripts that run UnrealBuildTool. So indeed, if you’re just doing an incremental cook, it won’t re-evaluate an unchanged Target.cs file and it won’t pick up on changes to external .uplugin files.

What is the type of build you’re making where you’re running into the issue? I.e. if you share the commandline or the editor action you’re using to start the build, it would be helpful for us to know. But, the short answer to this is “UBT needs to rerun”.

[Attachment Removed]

4

“In my tests on Friday I ran into it when doing local development workflow, doing a build of ‘development editor’ from Visual studio.”

Aha, got it. In the article I focused on versioning for the purposes of full, clean builds on build machines, but hardly included any notes on developer workflows. A detail I left out is that for editor targets on development branches we enable all plugins regardless of the plugin’s release version range. Realizing how that can add confusion, I’ll update the article to include the following notes.

  • We check the Target.Version.BranchName to decide whether we’re on a P4 development or release stream.
  • If that’s the prefix for a development branch, and we’re building an editor target, our implementation of ConfigureGameFeaturePlugins also simply enables all plugins.
  • If it’s a release branch, then only the plugins intended for that release version are enabled in the editor, to minimize discrepancies between available assets in the editor and the shipped build.
  • Build machines also only enable the plugins intended for that release version (the article’s snippets and my github project represent that).

Compiling all old and future plugin modules comes at a compile-time cost, but currently we just accept that cost.

Does that clarify the questions you had around dev workflows?

[Attachment Removed]

5

“Yep cheers, that clarifies things.”

Glad to hear that!

“Oh, and one other question, is there a reason for tracking the projectversion separately in Fortnite instead of just using ProjectVersion?”

No strong reason. The custom property allows us to store it in a separate INI file for a cleaner history, if someone would want to look at Perforce history to see when the versions got bumped without having to look at all of DefaultGame.ini’s history. It would have been trivial to use ProjectVersion.

[Attachment Removed]

6

“you pass in DisablePlugins to ConfigureGameFeaturePlugings but that’s then never used?”

Indeed! Not used in my snippet, not used in Fortnite. So you can ignore that argument, unless you want to take a different approach where plugins aren’t disabled by default.

“Additionally, there is a chance when printing OutEnablePlugins that it prints plugins that were enabled by other systems, should that print loop not act on a local list?”

Indeed. I believe the initial value for EnablePlugins is the optional -EnablePlugins=MyPluginA,MyPluginB that you can pass in when calling RunUAT BuildCookRun <myproject>. Those would be printed even though they weren’t enabled by ConfigureGameFeaturePlugins. You should indeed act on a local list if you only want to log what was enabled just via the release version logic.

[Attachment Removed]

7

Yeah, you should definitely remove some of the log statements for production or reduce the log level to something like Logger.LogInformation. I used LogWarning for non-warnings just to stand out from normal text in Windows Powershell, to illustrate but not to be used in production.

Target rules are expected to be evaluated multiple times, like per target, per build step (generate project files, compile). That part you can’t change, but you could perhaps change the verbosity level of the log. I’m not familiar with that - if it’s strongly desired, can you post a new question aimed at configuring BuildGraph or customizing actions triggered from UGS?

[Attachment Removed]

8

“Firstly, I’d like to say that ended up removing a lot of the log entries. Useful for understanding the system, but too verbose at the wrong times for something we’d actually deploy to developers/build systems.”

That is a sensible choice.

“There’s a crash waiting to happen in LexFromString, nullptr is passed in to ImportText for ErrorText and depending on what parser gets hit, on bad buffer data this can try to log a warning/error and hits a nullptr deref instead. I fixed this by passing in GLog which some other calls to ImportText do.”

Nice catch. I’ll make some notes from your feedback throughout these thread to update the article + example project.

“ApplyPrimaryAssetLabels can be called from other places side from ModifyCook, for example from UAssetManager::UpdateManagerDatabase. Is there any reason why the virtual ModifyCook itself didn’t get overridden? Or alternatively, at least only call the filtering logic in ApplyPrimaryAssetLabels during a cook?v”

ModifyCook() would have been a fine place to do it too. I believe one advantage from evaluating cook-rules in the editor is this:

  • In release streams, we apply versioning in the editor. This includes Target.cs files to disable out-of-season plugins, and these AssetManager primary asset labels.
  • This means that in the editor, when you right click a folder to Audit Assets, the Cook Rule column shows an accurate value.

It doesn’t provide benefit in development streams, if like us, you enable all plugins and all assets. But it’s useful for auditing in release streams.

“In ApplyPrimaryAssetLabels, “UE_LOG(LogTemp, Error, TEXT(” Asset ‘%s’ did NOT have a release version as asset tag!”), *AssetData.GetObjectPathString());" will be called for any asset that does not have a version range. I assume that is not actually desired?"

Up to you to decide. When I wrote that, I intended that you know exactly which primary asset types should and shouldn’t be versioned. Here I just chose for an ignored-types list ‘UnversionedPrimaryAssetTypes’ and assume that all other types are versioned by design. You can go the other way around, but I do think it’s valuable to define somewhere which types should have a version range and error if they don’t have the expected asset registry tags. Just in case someone introduces a subclass that overrides GetAssetRegistryTags() but forgets to call the Super:: implementation.

“This is more style, but I didn’t like having the separate `bHasSunsetVersion` property”

Sounds fine by me, whatever is developer friendly and not error prone. In the uplugin files I take it, that a missing SunsetVersion property is also interpreted as ‘never sunsets’?

[Attachment Removed]

9

“we were wondering if Fortnite actually uses FName based versions rather than integer”

The version can be numbers and special names like “Future”. The value can always be string-represented and be parsed into a USTRUCTS with drop-downs, that’s either a special name or version numbers. I’m not sure if we use FName or enums. Enums provide a better designer UX flow, but here what matters more is your own project’s needs - are you going to expose versioning fields to designers via UI? If so, then enums or gameplay tags make sense.

“Do you have special treatment of GFPs with a FortReleaseVersion marked as “Future” as with assets”

Yes, “Future” plugins gets always left out of distributed, Shipping builds. I’m not sure exactly about when and where that rule is implemented, but you can come up with many ways to check this, i.e. by build type (Shipping) or by release version (custom INI value, cmd arg, build job param). That choice should be up to you.

" you can have named releases (e.g., Release-Demo, and then in the Demo stream you just sunset any gamefeature plugin that you don’t want enabled in the demo)."

Absolutely, a special name for demo builds is a great use case for this.

“Is this something you’ve ran into or is Fortnite running with `bOverrideBuildEnvironment = true;`”

We use BuildEnvironment = TargetBuildEnvironment.Unique. It takes up more disk space if you use the same engine source with multiple game projects, but we don’t mind that.

[Attachment Removed]

10

Thanks for calling out the incorrect FMath::Sign<int8> call there. That’s just a mistake on my part.

“The potential misunderstanding, in development builds from the build machines, do you restrict the loaded plugins (and assets) in any way?”

We have the ability to make dev builds that include more plugins, and thus compile modules from those plugins, than will automatically be enabled at runtime. This is for testing purposes: there is a version range to cook and a specific version to load initially. The cooked plugins and their built modules can be enabled with commands, in development builds. I hope that clarifies.

I think that was primarily missing from my explanation, but not a blocker for you, is that right?

[Attachment Removed]

11

Thanks, some of your message aligns with our suspicions so good to see those confirmed. Will let you know where we get to with this.

One follow-up question. The logic in the target.cs files doesn’t run on any build, does Fortnite do anything special or because this is primarily for the build systems is it expected to only really operate on non-incremental builds? Otherwise I can see a version bump happen in the ini or env var but this not being picked up since the logic in the target.cs doesn’t run again. Editing the target.cs will of course cause it to be re-evaluated.

[Attachment Removed]

12

In my tests on Friday I ran into it when doing local development workflow, doing a build of ‘development editor’ from Visual studio. But I guess using the Fortnite workflow this would not actually be an issue unless someone is actually iterating on the release management logic:

  • Day to day devs will have their plugins set as ‘Future’ or as a specific version. The only time I can see someone run into an issue with it is if they locally restrict a plugin to a version earlier than the mainline ‘version’ in the ini files (or someone remote changes this and they sync). Or if the release version in the ini gets bumped, they p4 sync, and because target.cs doesn’t re-evaluate the game feature plugin is still enabled, even though it technically should not be. The current logic has no invalidation path (and there is no mechanism in UBT as far as I know to mark additional files to check for being modified to mark a build as potentially dirty? Otherwise the equivalent of defaultfortreleaseversion.ini could be added as such a dependency.
  • Less likely day-to-day, and this one is speculation, someone switches stream and while UBT detects source files are different, (and the version in the stream may be different), it doesn’t detect target.cs as having to be re-evaluated and thus plugins stay enabled/disabled when they should not be.
  • For our build system, we use C# buildgraph for current UE5 projects and XML for some older UE4 ones. Under the hood both the XML and C# workflow end up utilising the Compile not which then ends up execution the logic in UBTUtils.cs, which really just executes UBT. This should be fine for full builds, and even for incremental builds. Again, the only time I can see this go wrong on the build farm (would have to validate) is if it is an incremental build and the tagged version in the stream or environment has been modified. Target.cs wouldn’t re-evaluate in that case.
    [Attachment Removed]
13

Yep cheers, that clarifies things. I think it may still cause issues for people when they switch between release streams for local development, unless the stream switch somehow is guaranteed to re-evaluate target.cs. This is definitely a bit of an edge case.

Oh, and one other question, is there a reason for tracking the projectversion separately in Fortnite instead of just using ProjectVersion? I can imagine you may do this for ‘internal version’ vs ‘external version’ differentiation.

[Attachment Removed]

14

Thanks again. Noticed one further thing in the sample target.cs file, you pass in DisablePlugins to ConfigureGameFeaturePlugings but that’s then never used? I guess there is no real need since plugins are required to be disabled by default anyway? Additionally, there is a chance when printing OutEnablePlugins that it prints plugins that were enabled by other systems, should that print loop not act on a local list?

[Attachment Removed]

15

Thanks, pretty much there. One quality of life issue I just noticed is that when running generateprojectfiles ConfigureGameFeaturePlugins gets called dozens of times, so that is leading to a lot of output spam in UGS.

[Attachment Removed]

16

Thanks for all the info, I’m going to make this thread a little bit longer as we’re still running into some ‘real world gotchas’ when rolling out what we’re doing.

Firstly, I’d like to say that ended up removing a lot of the log entries. Useful for understanding the system, but too verbose at the wrong times for something we’d actually deploy to developers/build systems.

There’s a crash waiting to happen in LexFromString, nullptr is passed in to ImportText for ErrorText and depending on what parser gets hit, on bad buffer data this can try to log a warning/error and hits a nullptr deref instead. I fixed this by passing in GLog which some other calls to ImportText do.

ApplyPrimaryAssetLabels can be called from other places side from ModifyCook, for example from UAssetManager::UpdateManagerDatabase. Is there any reason why the virtual ModifyCook itself didn’t get overridden? Or alternatively, at least only call the filtering logic in ApplyPrimaryAssetLabels during a cook?

In ApplyPrimaryAssetLabels, “UE_LOG(LogTemp, Error, TEXT(” Asset ‘%s’ did NOT have a release version as asset tag!“), *AssetData.GetObjectPathString());” will be called for any asset that does not have a version range. I assume that is not actually desired?

Target.cs has some doubling up of logging with

```

    Logger.LogError("Failed to parse environment variable value {Arg0} into Major and Minor number.", EnvVar);

    throw new BuildException("Failed in GetReleaseVersionFromEnvVar()");

```

so I moved this into just the exception string and removed the separate logger.

This is more style, but I didn’t like having the separate `bHasSunsetVersion` property so in our code variant I’m using a sentinel version (basically set to max int) and am handling that as a special case. It has as a benefit it sorts into the ‘far future’ so is numerically a ‘never sunsets’ by default and also acts as ‘future’ in the same way.

[Attachment Removed]

17

It’s UAssetManager::ModifyCook which is already virtual, hence I was thinking that that may have been the better entry point. But audit assets is a good call, I’m going to have a look at that.

In the sample code, you loop through all primary asset types (and the majority of them will be unversioned) so the per-asset error message as-is will be confusing. I would probably look at a way to detect if a primary asset type is versioned and then only error for asset that for some reason are missing the version range (these would likely be legacy, unsaved assets).

And you’re correct, I’m handling ‘missing sunset version means never sunsets’ in uplugin files.

EDIT: I was having a discussion with someone else on this, and we were wondering if Fortnite actually uses FName based versions rather than integer. There are two benefits, as per DefaultFortReleaseVersion.ini you can have a nice allowlist of valid versions. You’d still parse them to integers for internal range comparison. The other is that you can have named releases (e.g., Release-Demo, and then in the Demo stream you just sunset any gamefeature plugin that you don’t want enabled in the demo). And, as with the fortnite ini, you could actually do the newer/older range checks based on position in the AllVersions array rather than numerically.

EDIT2: One thing we have ran into is that `BuildEnvironment = TargetBuildEnvironment.Unique;` puts the binary in a different location. And there’s part of the out-of-the-box unreal ecosystem that aren’t too happy about this (though it is a lot better in recent UE5 vs the 4.27 project I’m currently integrating it in). Is this something you’ve ran into or is Fortnite running with `bOverrideBuildEnvironment = true;`? Or I guess with the UE5 fixes Unique is ok.

[Attachment Removed]

18

Btw, I just came across https://forums.unrealengine.com/t/additional-clarifications-around-uplugin-exclusion-for-gfps/2655291/5 - that has a lot of good and useful information that probably should be included in the article. Especially the bits about gamefeatures project policies.

One additional question based on that. Do you have special treatment of GFPs with a FortReleaseVersion marked as “Future” as with assets, or is that not something you do and plugins are always explicit?

[Attachment Removed]

19

Hi, I’ve been holiday for a week so took me a bit longer to get back to you.

We have the system basically up and running for us now, thanks for your insight, I’ve come across a bug and I may have misunderstood you in one case.

Firstly the bug in the sample code. FExampleVersion::Compare uses FMath::Sign<int8>, which actually truncates the int32 versions. This leads to unexpected behaviour when your int value overflows int8. Better to just not use int8, and just use FMath::Sign (also makes it consistent with the C# code) unless you make the while pipeline int8.

The potential misunderstanding, in development builds from the build machines, do you restrict the loaded plugins (and assets) in any way? Or are all of them, including “Future” plugins, always enabled? Or, as per Additional clarifications around .uplugin exclusion for GFPs - #5 by ZhiKangShao, do build machines still filter plugins? Build machines are a bit weird here, because the build machine building the precompiled binaries on the development stream for UGS should act the same way as you do a local build, which means that all plugins should be enabled and thus this implies the build machine should not filter? But if the build machine doesn’t filter at all then that contradicts your statement of only cooking a version range. Alternatively, maybe you have a separate builder for pre-compiled binaries as you do for the editor binaries used by cook?

[Attachment Removed]

20

Fyi, for now we’ve given target.cs a commandline argument that our buildgraph execution that builds PCBs sets, so we can vary behaviour when it comes to plugin filtering.

[Attachment Removed]