Horde - no candidate changes after CL XYZ - build is not scheduled

Development Pipeline & Plugins
, ,
1

Hi,

we are using Horde server as our CD/CI and currently have 2 build templates that should be scheduled regulary. Both templates are based on default sample ue5-release-5.6.stream.json (incremental-build and packaged-build).

We just added schedules:

incremental-build is triggered when there is Code commit, only one build at once, maxChanges 0 and check interval is each 10 minutes

“schedule”: {

“enabled”: true,

“maxActive”: 1,

“maxChanges”: 0,

“patterns”: [

{

“daysOfWeek”: null,

“interval”: 10

}

],

“Commits”: [

“code”

]

}

packaged-build is triggered between 6:00 to 16:00 with 4h interval, only one build at once and requireSubmitted change

“schedule”: {

“enabled”: true,

“maxActive”: 1,

“requireSubmittedChange”: true,

“patterns”: [

{

“daysOfWeek”: null,

“interval”: “4h”,

“minTime”: “6:00”,

“maxTime”: “16:00”

}

]

}

From time to time incremental-build is not scheduled event when there is viable changelist, and when i look into horde scheduler log there is Skipping trigger of wil-main template incremental-build - no candidate changes after CL XYZ message.

This time issue hapened, i noticed that there were 980057 change incremental-build run and finished last time and then next day there is change 980231 and packaged-build started on this change (trigged by schedule), but few minutes later attempt to trigger incremental-build said that there is no candidate change after 980057 CL.

Im unsure why is this happening, because horde knows about changelist 980231 (packaged-build triggered) but when i looked into packaged-build log i see that it knows about 980231 changelist, but last code changelist is 980057 (image 20250612_packaged_build.png).

Also please take a look on 20250612_UGS_changes.png when you can see state of UGS changes.

We are able to solve this issue, by triggering incremental-build with previous code changelist manualy and somehow it solve this issue. Is there anything we are doing incorrectly?

I also attaching Horde server log Log20250611.txt from that day. At 14:00:41 there is log information that Packaged wil build is scheduled with commitId 980231 (marked as code commit in UGS) but there is another log lines that “code CL is 980057” so horde isnt recognizing this commit as code?

edit: then i tried to rebuild latest incremental-build configuration manualy, and noticed that build passed and was placed near change 980537 in UGS, but when i looked into horde log i noticed, that Last code change is old one 980057 instead of 980231. So we are in situation, when build looks like its from changelist XYZ but instead its build from ZYX, so users are confused. Take a look at attachment 20250612_horde_ugs_build_latest.png.

Thanks and regards

Lubos

2

Steps to Reproduce

3

Hi Lubos,

Can you provide some details on what the change actually is at 980231? Also, would it be possible for you to update your log configuraiton to be a bit more broad here? It’d be good to get debug logs from the server.

So just to disambiguate here, it seems that you’re referring to two separate (but potentially related issues):

  • commit & code commit should be the same, but aren’t (980537, 980057 listed, but should be 980537, 980537)
  • The schedule is not triggering on code changes

Let me know if I’m off in my interpretation here.

Julian

4

Hey there Lubos,

>thanks for your reply, can you please guide me how to configure logs to be broad enough for this case please?

This should cover how to update your appsettings in order to increase the log verbosity.

>About the change, it contains many .uassets and few .h and .cpp files (game plugin modifications). Unfortunatelly im not qualified to provide specific code examples. But are they matter?

I mostly care about *paths* to said files, than anything. Are there any patterns that appear when this occurs in the changelists? If so, we can try and emulate this in the test suite for ScheduleService.

Julian

5

Hey there Lubos,

Yeah that could certainly present some issues. If you detect this other scenarios share this pattern with multiple stream paths within the same changelist, I could see how there may be an issue in how we are parsing. You could be seeing code from one stream A, but the changes for stream B in said change are content only, leading to the scheduler not activating… but based on what you’re saying the //wil/main stream was what was visualized on UGS. Would it be possible to get a anonymized filelist for the changelist that failed? I wonder if we can place that in a ScheduleService test and step through the code to see what’s going on when we configure the schedule as you’ve listed.

Julian

6

Hey there Lubos,

I’ve reached out directly to you - no rush :slight_smile:

Kind regards,

Julian

7

Hi there Lubos,

No problem - always happy to help. Ah yes, this could very well be complicating things. Keep me in the loop - but just a heads up will be on vacation starting tomorrow and will be returning July 14th (Epic studio summer break starts June 30-July 11).

Kind regards,

Julian

8

Hey there,

Apologies for the delay here as we are just getting back this week (vacation was great!) and there is quite a backlog of things to get to.

I am not exactly sure what the best approach is here regarding the disambiguation between the two streams. For posterity, I spoke to one of the subject matter experts on this area and they gave some broader insights:

  • The intention of ugs marking things as Code is to facilitate the creation of PreCompiledBinaries
    • The behavior you are trying to avoid is:
      • [Code] [PCB] CL 1 - Modifies the data for Asset A
      • [Content] CL 2 - Asset of type A is saved/re-saved
      • [Code] CL 3 - Further changes to data for Asset A
      • [Content] CL 4- Asset of type A is resaved
    • You do not want to allow someone on CLs 1 or 2 to open the asset from CL 4 as it carries the risk of the editor crashing (and thus losing any non-saved, autosaved work)

[mention removed]​ Is there any guidance here on how UGS could be disambiguating this? I feel like this is likely coming from a stream X that has some import? Lubos can you confirm my hunch there? Because I *think* what you’re looking for is to have UGS discriminate what it considers a “badge-influencing(?)” file from a particular CL… but if there’s a stream import it could very well be the case that this is valid.

Regarding CodeFilter, I can’t seem to find any references to this within the UnrealGameSync.ini settings files that I’ve been looking at.

Julian

9

Hey there Lubos,

Sounds good - let me know if you need any pointers in the codebase for where to look. What’s most relevant will be the ScheduleService.cs (as I’m sure you’ve already found!). TriggerAsync is the best top of funnel to dig into (stream & template combo at the top, so it should help).

Kind regards,

Julian

10

Hey there Lubos,

So just thinking aloud this does make sense based off of how the import/share should work. Just reviewing the perforce documentation - it doesn’t seem like share should be excluded from the scheduler. The key thing about share is the bi-directional nature of coordination, in that if the parent has submissions into that path - it is considered a relevant change for the child, with the converse being true as well. Let me know if I’ve understood your comment appropriately.

Kind regards,

Julian

11

Hi Julian,

let me step into the conversation. It is not about the share but about the import.

The example lubos wrote:

import+ …

share Wil/Source/…

share Wil/Plugins/…

that is translated to this

//wil/main/… …

//wil/combat/Wil/Plugins/… Wil/Plugins/…

//wil/combat/Wil/Source/… Wil/Source/…

The import+ translates to //wil/main/… …

which is what triggers the change that should not be triggered. We will implement the fix for Horde and we can send you the changes so you can better see what we mean, regarding the UGS we will probably leave it with the bug and live with it for now.

TLDR: The problem is in the stream mapping, in Horde it ignores the type of a mapping, whether it is share, import, import+ or whatever, it just translates the path without context and includes it. At least that is what we found out.

12

Hey there,

Yes if you could send over a suggested fix we can look at integrating it. I do think there may be a bit of a preference here however. If a stream does import a folder, that means this stream will fundamentally be impacted by it any changes in that folder. Semantically, it would therefore make sense to kick of a build in the stream that is importing that folder.

Does that make sense?

Julian

13

Hi Julian,

I will try to make create some prototype of fix asap.

Yes, if a strema does import a folder, then it has impact on changes to that folder. But also there can be (and in our case is) is share in subfolder of that import path, and then its not so easy as you mentions. Because it can exclude changes from that folder.

And when this is combined with different way that UGS is resolving changelists, our users are a bit confused with that.

Regards

Lubos

14

Hi Julian,

thanks for your reply, can you please guide me how to configure logs to be broad enough for this case please?

About the change, it contains many .uassets and few .h and .cpp files (game plugin modifications). Unfortunatelly im not qualified to provide specific code examples. But are they matter?

And about issue(s), im not sure if it is same issue or not, but from my perspective, if Horde can recognize correct last changelist for Code submit, it will be okay, but im unsure what happens under the hood, and im not able to reproduce this issue properly to debug it. I only know it happens from time to time (this is not first occurence)

thanks and regards

Lubos

15

Hi Julian,

thanks for your advices, i will try to increase log verbosity as you suggested.

About the files in change 980231 i noticed something interesting today. The changelist contains data for two streams. In attachment you can find redcated screenshot of changelist 980231, where you can see that majority of changes was to //wil/combat stream and few files was to //wil/main stream. And all UGS data i provided to you was for //wil/main stream.

And if im correct, files that should change comit type to Code are on different stream, but UGS display Code type on main stream.

Maybe this can be the point?

Im unsure how it was in previous changelists, i will try to keep eye on it

Thanks and regards

Lubos

16

Hi Julian,

Thanks for the explanation — that makes sense and sounds plausible.

To be sure we’re looking at the same thing, would you prefer to receive the actual files from the changelist that failed, rather than just an anonymized file list? I can send them to you if you’d like.

Just let me know the best email address to use and I’ll forward them right away.

Thanks again!

Lubos

17

Hi Julian,

thank you for your efforts. I sent you information we talked about and few hours after that i noticed same behavior on another changelist. And from what i see it looks liek that it isnt issue on Horde, but in UGS changelist type recognition Code/Content.

I will examine this case more carefully and will let you know. Currently im looking into changelist that was only to //wil/main stream, but this change is displayed in UGS also on //will/main_release5.6_merge stream (stream that we are using to merge changes from upstream).

Stream is defined as

share …

import Wil/…

share Wil/Config/…

share Wil/Plugins/…

share Wil/Source/…

share Wil/Wil.uproject

import Tools/…

import Samples/…

and changelist contains submits 1 .cpp + 1 .h to //will/main/Engine/… that is shared on this stream (and few .uasets that are imported from main), but it looks like UGS takes this changelist as Code changelist and “waiting” for PCB to be build.

I will try to describe it more specific and will let you know.

thanks and regards

Lubos

18

Hi Julian,

I hope you had a great vacation.

I’ve looked into the issue a bit more and I’m now confident that the behavior originates from UGS, not from Horde. However, I’m not entirely sure whether it’s an actual issue or if we’re simply using it incorrectly.

What I can say for certain is that UGS classifies a changelist as a Code Change if any file in the changelist is recognized as a CodeFile. By default, the logic to determine whether a file is a CodeFile is based solely on the file extension—not the stream it belongs to.

This becomes problematic when using multiple streams based on the same codebase: UGS marks the change as a Code Change for all streams, even though Horde does not trigger a build marked as Code in these cases.

The relevant method is:

UnrealGameSyncShared -> Utility.cs -> ~line 69 -> IsCodeFile(string depotFile)

While exploring the UGS code, I also noticed that this behavior can be customized via the UnrealGameSync.ini file—specifically, through the [Perforce] section and the CodeFilter property. However, I wasn’t able to find any documentation on this.

So my questions are:

  • Is this default behavior intentional and correct?
  • Could you point me to any documentation or provide guidance regarding the CodeFilter setting in the .ini file, if this is the correct way to configure it?

Thanks and best regards,

Lubos Suk

19

Hi Julian,

I believe i understand the need for the Code tag and the PCB blocking behavior you described.

I can also confirm that the issue appears to be related to stream importing. However, from my perspective, UGS doesn’t seem to take the stream into account when determining whether a change is considered “Code” or not. The logic that identifies code changes seems to rely solely on file extensions. That’s why my initial idea was to use filters. It’s a shame you couldn’t find any references, but I can revisit that approach later if no better solution is found.

I noticed that UGS can recognize a CodeFilter setting in UnrealGameSync.ini like this

[Perforce]

CodeFilter=//wil/main/…

While you were on vacation (glad to hear it was great! :)), I was investigating an issue where some schedules in Horde were being triggered too frequently (first I complained that builds weren’t triggering — now it’s the opposite! :p). I’m currently trying to understand the part of the code that translates P4 changes into the MongoDB cache. It might be using the same logic, and I’ll let you know if I find anything relevant.

Regards

Lubos

20

Hi Julian,

it seems i already figured it out. And it realy seems that there are two separated issues (from my point of view). One for UGS and one for Horde.

UGS - as i already mentioned im sure, that in UGS the stream mapping is not take into account when determining to which changelist it belongs to. So there are incorrectly marked code changes in UGS and blocking data download.

Horde - Thanks for pointing to ScheduleService.cs but this part of code looks okay for me. I tracked it to the part when data from P4 are replicated to MongoDb (PerforceServiceCache.cs). There i found code that is resposible for resolving stream mapping, but its handling import mappings incorrectly. Because by combining import and share, some paths can be excluded form parent stream, and this code is not taking this into account. It takes every record from stream View and marks it as Include or non Include (based on -prefix created by exclude), and everything else is marked as include.

So when there is a stream with view (that has parent stream)

`import+ …
share Wil/Source/…
share Wil/Plugins/…

that is translated to this

//wil/main/… …
//wil/combat/Wil/Plugins/… Wil/Plugins/…
//wil/combat/Wil/Source/… Wil/Source/…`

then submit into //wil/main/Wil/Plugins/XYZ is considered as comit into child stream, that is not correct because share overrides this. Or am i wrong?

So i miss there some advanced stream mapping resolver, or it can be handled by p4 where command.

Thanks and regards

Lubos Suk