Recently I have setup a automation script to mirror the epicgames perforce stream to a github repo, while maintaining all the binaries and source code necessary.
This is quite large unsurprisingly and was looking for an alternative.
My understanding is the following:
-For policy reasons, I need to mirror the perforce stream
-The perforce stream contains all required binaries for the UnrealEngine
-On github, there is a gitdependencies file, which setup.bat/sh utilizes to pull these binaries.
-This dependencies file is not on the perforce stream and is generated at some point after that, based on the perforce contents.
If my understanding is correct what I would like to know is if there is a way to mimic this same pipeline myself, so that I can comply with mirroring the P4 stream to github, while keeping it lean, and functional via setup.sh and gitdependencies built off whatever version of UE we are pulling from the P4.
The GitDep system is based on internal tools that are not available to licensees. Which stream are you mirroring? It would be better to fork one of our already branch from GitHub as it is already configured to download the binaries separately.
Currently the 5.7 stream, potentially moving to 5.8 soonish. For reasons outside of my control we cannot fork the github, which is why we were told to use the perforce, and in setting up a mirror of the P4 stream onto a internal github repo, I wondered about the gitdep system. If there is no way to get that as a licensee, is there perhaps some form of gitignore + gitattributes that properly contains and marks the binaries from p4 that is evergreen?
Just looking to see if I can better close these gaps, as our internal git repo seems to be inconsistent between users.
Those git configs found in Engine/Build/Git are the ones for an ignore that depends on dependencies and setup correct?
So then either we find a way to get the github hosted versions of gitdeps for the associated P4 CL, or we keep doing what we recently did and continue hosting all the binaries ourselfs via git lfs.
theres no way for us to create git deps as licensees. Out of curiosity would it be possible to get the gitdeps added to the p4 stream instead?
I’m not GH expert but that would be my guess. This is what we are using in our GH repos and anyone that forks the repo will use them.
Finding the right gitdep file should be easy. The rate of change for this file is fairly low and the GH commit comments contain the corresponding Perforce CL. In the case of 5.7, that file will not change since we have moved on from that version.
We just happen to still be on 5.7 at this time, we plan to migrate to 5.8 in the near future, is it expected for the gitdeps to change much over 5.8s lifecylce?
Again if the gitdeps could also just be pushed to the p4 stream as needed that would resolve most of this issue for us.
The gitdeps file for 5.7 is static at this point. We will not change the stream at this point.
On the 5.8 side, we are still working on hotfixes so the file will keep requiring updates as we are iterating. The file is only updated when we submit a binary file on P4. The commit comments contain the CL number so it is possible to reconcile the 2 sides. It is possible to find the cutoffs for the hotfixes by looking at the history of the Engine folder. There will usually be a couple of “Localization Automation” updates when we are about to release. Those happen over a couple of week where the stream is closed except for last minute fixes.
I discussed with the owner of the replication of our stream to GH and we can’t add the gitdeps files to the streams. The files are generated as part of a system that manages the replication to GH which currently reads from Perforce but doesn’t write to it. Adding writing would incur significant changes to the system. There is also a delay between the submission in Perforce and the replication to GH, ie generation of the gitdeps file. There is no way to keep consistency of the file for a given state of the stream as the Gitdeps will always be ready a couple of hours after the original change.