Hello!
We have had a lot of issue with our Perforce server lately. After searching and exploration, it seems that’s Horde is opening and keep idling connexion every minutes without closing it, making our Perforce server reaching the maximum configured connections amount.
Do you have already have this problem?
We are using Horde with Perforce Password on config file, not Perforce Ticket and have deployed the 5.8 version on Horde.
[Attachment Removed]
Hello! The only open issue I can find that might be relevant is a case in which using incorrect credentials can cause the Horde Agent to accumulate idle threads and open connections (UCS-21201).
This affected another licensee; the cause was a user updating the LDAP credentials, but not the credentials Horde was using.
There’s no direct indication that this is the same scenario, though there could be another issue triggering similar behavior.
Could you please share the following that the team would need for further investigation?
- Your physical Horde deployment setup (number of servers and agents).
- Your Perforce setup (physical, number of streams, server max commands limit, etc.).
- A capture from Perforce showing active processes and long-running tasks using `p4 monitor show -ael`.
- Ideally, this shows the growth in real-time, but it could be worth taking two captures some time apart.
- Horde logs from the server’s logs directory, covering the same window as the capture above.
If you are able to raise the log level, that would give us additional useful information.
E.g., the `Unable to login` message is an error, but `Unable to allocate existing pooled connection` is on the debug level.
You can add `“HordeServer.VersionControl.Perforce”: “Debug”` to the existing Serilog.MinimumLevel.Override block in your Horde\Server\appsettings.json config to enable only these messages.
[Attachment Removed]
Hi!
The issue might be comming from issues with DNS. We have sometimes DNS issues and the Horde issue occured since this issue.
As our setup, we use one server hosted in a docker container and 15 agents running on Windows with dedicated computer.
We have one VM dedicated for Perforce, with 15 streams opened in Horde. And the limit is 2500, it’s the limit that we are reaching.
The capture from Perforce showed dozen of Horde tasks, opened for more than 3 hours (sometimes 6 hours), in an Idle state
I will try to gather informations about the growth, but the issue sometimes append during night, and it’s growing really fast, so no growing on the day.
I will try tomorrow to add our perforce server directly as an extra host on Docker, and increase the log level in the same time!
[Attachment Removed]
Hello!
Quick followup on this issue. I think I’ve found the problem.
We have 2 Horde server on our environment. One for production, and one for testing before production. We have changed our Perforce certificate not long ago, and we forgot to update the test one.
So the Test server keep trying to connect to Perforce, fail due to the change of certificate, and never release the connection after all.
This might be a bug that you have already fixed, the test server wasn’t on the latest Horde release I think.
[Attachment Removed]
Hello! That does sound like it could be linked to the issue that I mentioned before.
Were you able to pull the logs from the test server that was attempting to connect to Perforce?
If so, I can attach those to the internal ticket and try to get the priority bumped. Thanks in advance!
[Attachment Removed]
Hello!
This is an example of logs that we had before I fix the issue on our test server.
Sadly, I didn’t raise the log level for this container (because I wasn’t considering it before finding the issue was from there)
[Attachment Removed]
Thank you for sending these logs over; it doesn’t look like we have this specific issue tracked yet, so I’ll create a ticket for it.
The good news, in the meantime, is that these logs do already mention the P4 trust resolution step.
[Attachment Removed]
Yes, that’s from this log I’ve found the issue. But I can’t find why when there is a certificate issue it keep opening new perforce connections without clothing it
[Attachment Removed]
It looks like we’re not disposing of the connection correctly if/when LoginAsync() throws within ConnectAsync() (PerforceConfigSources.cs).
I’ve left the further investigation and fix with the team, but it may be a case of adding a try-catch around the login, calling dispose on the connection in the catch block.
[Attachment Removed]
I just wanted to let you know that this has been fixed in CL 56824258.
https://github.com/EpicGames/UnrealEngine/commit/8cec04cfaed75335466bfaf48c46c672f8721cf6
[Attachment Removed]