From the AutoSDK link, it is stated that more information can be found in the Readme for how to setup an SDK for each platform, however the Readme for Linux is empty, it simply states: “Details forthcoming.”
AutoSDK is a repo of SDKs and their compilation tools so that installing them is not required. You should not need Turnkey in this case as the Linux toolchain does not need installing other than what setup.bat is doing. Turnkey’s goals are to facilitate the distribution and installation when some registering is required on a programmer or artist machine. An example would be when a Visual studio extension need to be installed or other tools that have to live under Program Files on a Windows system.
Our build farm does use AutoSDK and we rely on a Perforce Depot that contains all current SDKs for our target platforms. Our build script are synching this depot in a common local folder on each machine in the farm. Any SDK change in the Perforce depot will be picked up by our build machines on their next synch. It could also be possible to use a network share to host the official AutoSDK folder but it is important to copy the data to a local folder for performance reasons.
One important point is that we recommend placing the AutoSDK folder outside of the Engine’s hierarchy. The Engine\Extra folder can be misleading but is simply an example of the structure the folder should have. One of the goal of AutoSDK is to share the tools between multiple copies of different projects that could be using different engine version potentially targeting different SDKs. Something like C:\AutoSDK (or any other local drive) is a common location to set it.
Regarding the Linux cross compile toolchain, HostWin64/Linux_x64/v25_clang-18.1.0-rockylinux8/ is the proper location and you can reuse the Setup\unsetup batch files from older Linux toolchains that you identified. I validated that the version we are using internally has not changed.
So in your case, you only need a central location from where the build machine and users can retrieve the current SDKs required by the Engine\Project.
Thank you for the detailed explanation and sharing how the Build farm is utilising AutoSDK at Epic.
I had a follow up question though, you state “…the Linux toolchain does not need installing other than what setup.bat is doing.”
When I wanted a Win64 build agent of ours to produce a Linux server build, I needed to install the below indicated exe on the build agent. Only then did Unreal successfully recognise Linux as a platform it can build.
[Image Removed]
So does this mean that if I only run setup.bat on a build agent, it will be able to successfully produce a Linux build on a Win64 host? Are no Linux files required to be present in the AutoSDK repo that the build agent would sync to its local system?
We will be hosting our AutoSDK files in a Perforce depot as well, so I am trying to figure out which files should be present in the Linux folder of the AutoSDK repo.
The full Linux toolchain does need to be under: HostWin64/Linux_x64/v25_clang-18.1.0-rockylinux8
One person needs to install the toolchain on a PC and move it to the right place in order to update the AutoSDK repo.
Setup.bat does create the environment variables required for our tools (mainly UBT and UAT\Turnkey) to see and use the toolchain. This is done inside the process and its temporary to its duration. This part is also dependent on the target platform and its tools.
Once you have your central repo set, configuring a new users or build machine involve the following:
Sync the AutoSDK repo to a local drive
Create the UE_SDKS_ROOT environment variable and point it to the local AutoSDK folder
Thanks for the input, that helped to stage the correct SDK files, as I created an Install.zip file which contains the below and is structured as follows:
Install.zip
v25_clang-18.1.0-rockylinux8 files that are installed via the exe installer
Setup.bat
Unsetup.bat
I am getting the following error when I run RunUat.bat turnkey and attempt to see if the Linux AutoSDK files hosted in our perforce are being correctly recognised:
Turnkey exception: Unable to find stream spec in perforce operation //depotname-sdk/main/Linux_x64/*/Install.zipBelow is a screenshot of the options I am selecting to get to that error:
[Image Removed]The main Turnkey manifest that points to the other Turnkey manifests is located in Engine/Build/Turnkey with this line:
<Manifest>file:$(EngineDir)/Platforms/*/Build/Turnkey/TurnkeyManifest.xml</Manifest>Below is the Linux Turnkey manifest file located at Engine\Platforms\Linux\Build\Turnkey with the below content:
`<?xml version="1.0" encoding="utf-8" ?>
Linux
AutoSDK
$(ExpVersion)
Linux Toolchain
fileexpansion:perforce://depotname-sdk/main/Linux_x64/$[ExpVersion]/Install.zip
`Are you please able to shed any insight into why Turnkey is throwing the exception mentioned above?
I get it whenever I try to search for the Linux SDK using Turnkey.
As far as I can tell, it should be setup correctly and chained together correctly via the xml manifest files. It is clear that Turnkey is able to reach the P4 depot set up to host the SDK files.
I have ensured that UE_SDKS_ROOT is created and pointing to the local AutoSDK folder and that it is synced to the local drive.
I just wanted to provide an update as I have been troubleshooting further on my side, and made some changes and fixes on my side. This helped to produce a different Turnkey exception.
I fixed in the Linux specific Turnkey Manifest file the perforce path to include HostWin64. I had fixed this in the depot itself but forgot to alter it in the manifest file.
I changed the square brackets in the file expansion line to be round. See the updated manifest file below:
`<?xml version="1.0" encoding="utf-8" ?>
Linux
AutoSDK
v_$[ExpVersion]
Linux Toolchain
fileexpansion:perforce:\\depotname-sdk\main\Host$(HOST_PLATFORM_NAME)\Linux_x64\v_$(ExpVersion)\Install.zip
Win64
`I also fixed and updated the naming/structure in the depot to be (had a duplicate main folder), so it is now as below:
Turnkey exception: Bad expansions output from CopyProvider (perforce:\\depotname-sdk\main\HostWin64\Linux_x64\v_$(ExpVersion)\Install.zip returned 0 count, expected 1, from perforce:\\depotname-sdk\main\HostWin64\Linux_x64\v_$(ExpVersion)\Install.zipI have looked through the copy providers section in the Epic docs (https://dev.epicgames.com/documentation/en\-us/unreal\-engine/setting\-up\-turnkey\-for\-your\-organization\-in\-unreal\-engine\#copyproviders) but I have not located what could be the issue. As far as I can tell comparing the examples provided by Epic and the docs this should be correct.
I have tried using forward and back slashes, square and round brackets in the manifest file, and replacing the 3 dots with Install.zip.
Could you please provide some guidance on what could be going wrong?
Thanks for the help! I have fixed that to the below and it is now giving the same error as before my last post above.
`Turnkey exception: Unable to find stream spec in perforce operation \depotname-sdk\main\HostWin64\Linux_x64\v_*\Install.zip``<?xml version="1.0" encoding="utf-8" ?>
Linux
AutoSDK
v_$[ExpVersion]
Linux Toolchain
fileexpansion:perforce:\\depotname-sdk\main\Host$(HOST_PLATFORM_NAME)\Linux_x64\v_$[ExpVersion]\Install.zip
Win64
`Given that the wildcard is being correctly substituted in. I am suspecting that the Perforce depot is setup incorrectly?
A mainline depot has been setup and that is what I am requesting Turnkey to resolve. Is a specific stream required to be created off of a mainline depot? Or have I created the wrong type of depot completely?
Still no luck unfortunately, getting the same error with the below in the manifest file for Linux:
`<?xml version="1.0" encoding="utf-8" ?>
Linux
v_$(ExpVersion)
Linux Toolchain
AutoSDK
fileexpansion:perforce://depotname-sdk/main/Host$(HOST_PLATFORM_NAME)/Linux_x64/v_$[ExpVersion]/Install.zip
Win64
`I have tried forward and back slashes but the same error is given.
Turnkey exception: Unable to find stream spec in perforce operation //depotname-sdk/main/HostWin64/Linux_x64/v_*/Install.zip
The one troubleshooting thought that came to mind was to hard code everything in the Manifest file and see if Turnkey can at least find the SDK files correctly in Perforce. Below is the manifest and the error it gave.
`Turnkey exception: Bad expansions output from CopyProvider (perforce://depotname-sdk/AutoSDK/HostWin64/Linux_x64/v23_clang-18.1.0-rockylinux8/Install.zip returned 0 count, expected 1, from perforce://depotname-sdk/AutoSDK/HostWin64/Linux_x64/v23_clang-18.1.0-rockylinux8/Install.zip``<?xml version="1.0" encoding="utf-8" ?>
Linux
v23_clang-18.1.0-rockylinux8
Linux Toolchain
AutoSDK
fileexpansion:perforce://depotname-sdk/AutoSDK/HostWin64/Linux_x64/v23_clang-18.1.0-rockylinux8/Install.zip
Win64
`It looks like something may be going wrong else where but I am not sure where there may be additional logging.
1- Our Regex parser doesn’t like the name of your depot, ie depotname-sdk, and stumbles on the dash…
2- When using a fully expanded string, you are running into an early exit of the PerforceCopyProvider.Enumerate which triggers the exception…
I’ll discuss with the owner of that code so those bugs are resolved in a future release. In the mean time, I recommend that you rename the depot so it only uses letters.
Thank you for the help! I can confirm that removing the dash and moving back to using the expansions, allowed Turnkey to successfully read from the Perforce depot and show the files that are present in the depot for the Linux AutoSDK.
One minor thing I have noticed is that the below line
<HostPlatform>Win64</HostPlatform> Does not cause Turnkey to register that it is a Win64 Host via the UI. Only when I put the host platform within the Source block (see code snippet below), did it register.
<Source HostPlatform="Win64" CopyOverride="perforce://depotnamesdk/main/Host$(HOST_PLATFORM_NAME)/Linux_x64/$(ExpVersion)/...">Here are screenshots of the output of Turnkey CLI showing this behaviour
[Image Removed]
One additional note in the screenshot above, is the yellow line under AutoSdk. I needed to change the spelling to exactly AutoSdk and nothing else in order to get Turnkey to register this as an AutoSdk file. Hopefully this will help anyone else in future.
Another update, I tried to get Turnkey to setup the Linux AutoSdk. However Turnkey is not seeing it as a valid AutoSdk to install. See the below output from Turnkey
[Image Removed]
I do not see the Install.zip being extracted anywhere and Setup.bat being run. No new txt files or Environment variables were created.
I created the Install.zip by zipping the entire contents of the folder to which the Linux Tool Chain Exe Installer installs to (default is C:\UnrealToolChains). I also made sure that Setup.bat and Unsetup.bat were added to the zip archive.
This was my understanding of how to create the AutoSdk files for Linux. Is this correct?
Our AutoSDK Perforce repo contains the already installed SDKs. The synched data is used from there through the AutoSDK env var mapping. In your case, that would mean UE_SDKS_ROOT pointing to d:\P4\XXXsdk\. Setup\Unsetup would be executed from there through UBT.
I thought that Turnkey would consume the Install.zip file on sync but it appears that this is only for the non-AutoSDK cases.
Thanks for the help. I have checked the installed files into the repo as you have indicated. It appears I have been successful at last to install the Linux AutoSDK
Two strange things to note though:
1) Turnkey still does not register it as a valid SDK in the list for Linux. But it does display under the Invalid list.
2) If I tell Turnkey to install the Invalid SDK, it now does so successfully. Turnkey even registers that there is an Installed AutoSDK.
The screenshot below show this behaviour in the command line, on the left the yellow line indicates the installed AutoSDK while on the right the red line shows the same variable empty prior to the install.
[Image Removed]
Both show that the Linux AutoSdk is not valid, yet I can see that setup.bat was run, as the output text files the setup generates are now present as seen in the screenshot below:
Thanks for the advice on checking via the Editor and compiling once after the AutoSDK was updated. I can confirm that the AutoSDK is being recognised in the Editor and via the Turnkey commandline. I am producing a server build using the editor to further verify but I would assume this will work.
[Image Removed]
I really appreciate all the help in getting AutoSDK setup for Linux.