Horde - Multiple perforce clusters with autosdk streams configured cause an error during conform

Development Pipeline & Plugins
, ,
1

Hi,

We are setting up a horde server. We have a sample project on a staging perforce instance, and a project on a production perforce instance. We have set up horde to have 2 perforce clusters each with an autosdk stream configured.

When running a conform, it fails with `Unhandled exception while running conform: Failed: Perforce password (P4PASSWD) invalid or unset. (Generic=Config)` and seems to be when syncing the autosdk.

Each of these perforce servers work individually as long as they are the only one, but as soon as we have both, it fails. I did notice that for the autosdk it uses p4.exe instead of the native client, and I found a ticket suggesting this may perhaps be a known issue: [Content removed]

I’m not sure if it relates specifically to the issue we are having, but perhaps there’s an issue with using the p4 executable and syncing two different autosdk streams that each use separate tickets.

We are trying to avoid editing Horde code as much as we can, and was wondering if there was some setting we are missing that would help us with this use case.

This is an example snippet of how we have the perforceClusters section set up:

"perforceClusters":
[
    {
        "name": "unreal-prod",
        "servers": 
        [
            {
                "serverAndPort": "unreal-prod:1666"
            }
        ],
        "credentials": 
        [
            {
                "userName": "horde-build",
                "ticket": "12345"
            }
        ],
        "autoSdk":
        [
            {
                "name": "autoSdkWorkspaceProd",
                "stream": "//autosdk/main"
            }
        ]
    },
    {
        "name": "unreal-stg",
        "servers": 
        [
            {
                "serverAndPort": "unreal-stg:1666"
            }
        ],
        "credentials": 
        [
            {
                "userName": "horde-build",
                "ticket": "56789"
            }
        ],
        "autoSdk":
        [
            {
                "name": "autoSdkWorkspaceStg",
                "stream": "//autosdk/main"
            }
        ]
    }
]

[Attachment Removed]

2

Hello! Thank you for the detailed write-up, investigation into other cases, and for sharing your config.

It looks like you’ve run into a limitation on our end with ticket-based auth, not a config issue per se.

When a ticket is supplied for a cluster, the Horde agent applies it by setting the global P4PASSWD environment variable.

During a conform, the agent sets up each workspace in sequence, so with two clusters, the second cluster’s ticket overwrites the first.

Because the AutoSDK sync uses the p4.exe client, the environment variable is used, which causes this issue.

Support for this setup (multiple Perforce clusters, multiple AutoSDKs) has been raised in the past, but it is not how our build system is used internally.

At Epic, we use a build machine password and not per-machine tickets, which is stored as a secret (see the Secrets docs: https://<horde_url>/docs/Config/Secrets.md).

P4 accounts are service accounts set up to impersonate users, and flagged as such in the config (“canImpersonate”: true).

Additionally, we segregate agents so that each agent/pool services only one Perforce cluster.

This approach is a recommendation we have given to licensees in the past and has resolved issues related to ticket-based auth.

Would it be possible to switch your agents to password-based auth?

[Attachment Removed]

3

Hi, thanks for the response.

We were hesitant to use passwords and went the ticket route because the documentation seemed to suggest that stored secrets were meant to be ingested by build graphs rather than the configs themselves. After you suggested using a secret, I dug around in the code and realized there actually is support for secrets in the configs.

After setting up the secret provider, we were able to use the secret using the `horde:secret:` prefix in the globals.json. Once using password-based auth, this indeed solved our issue.

Just for future reference if anyone else searches for this issue, we set up our Horde to grab the perforce passwords from Vault with the following setup:

{
    "plugins": {
        "secrets": {
            "providerConfigs": [
                {
                    "name": "VaultProvider",
                    "provider": "HcpVault",
                    "hcpVault": {
                        "credentials": "presharedkey",
                        "endpoint": "https://<vault-url>",
                        "preSharedKey": "<Vault token>"
                    }
                }
            ]
        }
    },
    "secrets": [
        {
            "id": "perforce-secrets",
            // Some values read from HashiCorp Vault
            "sources": [
                {
                    "providerConfig": "VaultProvider",
                    "path": "/v1/horde-secrets/data/perforce",
                    "format": "json"
                }
            ],
 
            // Only allow Horde agents to access this
            "acl": {
                "entries": [
                    {
                        "claim": {
                            "type": "http://epicgames.com/ue/horde/role",
                            "value": "agent"
                        },
                        "actions": [
                            "ViewSecret"
                        ]
                    }
                ]
            }
        }
    ]
}

{
	"plugins": {
		"build": {
			"perforceClusters": 
			[
                {
					"name": "p4cluster",
					"servers": 
					[
						{
							"serverAndPort": "perforce-server:1666"
						}
					],
					"credentials": 
					[
						{
							"userName": "service-account",
							"password": "horde:secret:perforce-secrets.password"
						}
					],
					"autoSdk":
					[
						{
							"name": "autoSdkWorkspaceProd",
							"stream": "//unreal/autosdk/main"
						}
					]
				}
			]
		}
	}
}

[Attachment Removed]

4

You’re welcome! It looks like we’re missing documentation for this usage and syntax on our Secrets page, so I’ll update that.

https://<horde_url>/docs/Config/Secrets.md#using-secrets

Using secrets from config is definitely supported and recommended. Config properties decorated with “[ResolveSecret]” can leverage this.

Currently, this applies to PerforceCredentials.Password, PerforceCredentials.Ticket, TokenConfig.ClientId, & TokenConfig.ClientSecret.

[Attachment Removed]