The source of the problem seems to be using IOptionsSnapshot with a Singleton. Per the MS documentation for IOptionsSnapshot, “Is registered as Scoped and therefore can’t be injected into a Singleton service.” But, for example, HordeClient uses it during the dependency injection process. I was able to get things working again by converting several IOptionsSnapshot to IOptions. It loses the updating function of course, but it doesn’t exception. Is this a known issue? I was a little confused when it happened. It works fine if you run as local because the checking isn’t done in that case. But the packaged one breaks. Additional question, we’re new to horde - we’re not doing something wrong with it are we? (That’s causing this for us)
Steps to Reproduce
- Sync 5.6
- Run BuildHorde.xml
- Deploy
- Install the agent as service
- (Optionally) Download zip and launch
It seems to install, but on start of the service, it exceptions leading to a timeout starting. If you run the agent after unzip, you can see the exceptions.
Hey there,
This is not a currently known issue - and this is the first time I’ve seen this reported. Internally we use Docker images, so it’s possible there could be something amiss. I’ll check internally to see if there’s any notes on this.
Kind regards,
Julian
Hi there,
My apologies on this delay!
We haven’t see any such occurrence of this on our internal installers for the 5.6 agent. Some of the feedback from one of the SMEs was:
- Could there be multiple clients running?
- Are the permissions not correct for the particular agent
Out of curiosity, do you see any patterns on the agents where this is occurring, or is it a 100% repro? I have been running 5.6.0-43060725 on my internal dev setup for the past couple weeks now with no observances of this. I am using the bundled installer from the corresponding Horde Server (5.6.0-43060725).
For posterity, this is the only report of such an issue that I’ve seen in the public space, so I wonder what other environmental factors could be at play here in your local setup.
Kind regards,
Julian
Hey there,
Thanks for sharing the zip of the code. Do you have some reference points regarding your divergences? FWIW my project also has the DOTNET_ENVIRONMENT - Local for agent debug settings (which I don’t believe should impact your packaged build in any way).
Furthermore, this looks like an agent issue, so if your divergences are constrained to Server I don’t see how it’d impact either.
Kind regards,
Julian
Hey there,
Ah - so I was just chatting with one of my colleagues regarding this, and he did note that ValidateScopes=true appears to be set in dev environments, but false in prod/others. This is where you’re likely hitting these issues, as it aligns with the divergences that you’ve had to make on your end (force development - which in turn forces ValidateScopes).
For posterity, as similar discussion on the subject within Github: Why I can inject Scoped service into BackgroundService if not in development environment? · dotnet/runtime · Discussion #109491 · GitHub
Related, my colleague pointed me in the direction of a recent change he’s made that could sidestep this issue:
- https://github.com/EpicGames/UnrealEngine/blob/ue5\-main/Engine/Source/Programs/Horde/Plugins/Build/HordeServer.Build/BuildConfig.cs\#L156
- Specifically https://github.com/EpicGames/UnrealEngine/commit/b0c73e3fb6055cd84e1ec7c0a2e4f1d436366376
I’ll create a ticket locally for us to investigate the root cause. In the interim, you *should* be able to strictly set the ValidateScopes to false:
// Wherever we BuildServiceProvider we encorce this await using ServiceProvider serviceProvider = services.BuildServiceProvider( new ServiceProviderOptions() { ValidateScopes = false});Kind regards,
Julian
Has there been any discovery on this matter? I’m not sure how we would have caused the issue, but it’s always possible, especially with systems this large/complex.
I’ve tried to attach a zip of the code sort of surgically removed from the codebase. For me, it repro’s 100%. I know we made some changes to make the appsettings files work correctly for the different environments. It doesn’t seem like that should affect this, but I guess it’s possible. We also had to introduce a hack because our perforce server was setup wrong in the distant past. You don’t want to know, lol. But overall, I don’t think we caused it.
Make sure this setting isn’t involved - microsoft built it to skip the checks that exception for us.
[Image Removed]
Our changes have affected the agent as well as the server, but as far as I know the entirety of the changes were to force the p4 host value to be the empty string and to respect the environment setting wrt the app settings, i.e. - with dotnet env set to development use the development appsettings. Those were for our weird p4 server setup and what we thought was an oversight but we needed to support our dev/main/release setup for Horde development. You should be able to repro the issue, I think, if you just change that local->Development in VS and then debug. Hope that helps.
Strangely, that didn’t help for me. But looking at some articles and after some testing I did find that changing (in ServiceRun.cs)
IHostBuilder hostBuilder = Host.CreateDefaultBuilder() ==to==> IHostBuilder hostBuilder = Host.CreateDefaultBuilder().UseDefaultServiceProvider(options => options.ValidateScopes = false); does succeed with the original source code. Not sure what’s different that makes our behavior different. Just wanted to update, no action to take at the moment.