Source Control disenchantment

Hi guys!

I have been developing a game with some guys for like 2 years now. We have been using git + git lfs with bitbucket server (10 user) for quite some time.
Sadly we came to a point where stuff is just getting worse and worse.

Our repo is at like 160GB currently and lfs has had a lot of problems since day one.
It seems that lfs is not designed for a large repo like ours which has tons of binary files.
We had issues even enabling lfs in the beginning and worked around those installing git lfs by using “git lfs install --skip-smudge” which forced us running our git lfs pull manually but at least it worked.
Sadly not anymore. It seems that cloning the repo for new users kinda sucks because the download speed does not even rise to half of the bandwidth and the server times out after like 1,5gb.
Even after that the repo is completely broken because there are still quite a lot 0kb files.

It seems that the only other alternative is perforce, but they are ****ing insane.
I mean who could afford this price? The 5 user limit is a joke to me. We are doing good with the 10 user limit by bitbucket, but 5 users cmon…
The next pricing step is impossible for us as a hobby team.

Do you guys have an idea what else might be worth a thought to consider as a new source control solution (not gitlab, gitlab was even worse than bitbucket)?

Cheers!

The only real option is perforce. There is just no way around it - git lfs is really bad.

They only allow 5 users, but they allow 20 workspaces. So we just have people share a user and identify them by their workspace instead :smiley:

Prior to switching to Perforce we were using Subversion.
UE4’s subversion support has no doubt improved since then.
Perhaps Subversion is worth a look?

Annnnndddd the other shoe drops. :wink:

We have been running an Internet based project for years and I can confirm that source control over the InterWebs is a Myth once the need for art content becomes involved and in general art content does not like GIT at all period. With GIT you have to do this that and the other thing just to get it to work and as you found out the more you add the more miserable things become just to share some project files.

We started out using git and all that fun in-app source control but we discovered real quick some major issues that there is no “single” solution that would make life easier and in no particular order over time this is whats going to happen based on a single solution.

  1. In app source control loves to lock out project files that does not need to be locked out as part of a team effort. For example coders can do their coding thing while content creators can iterate the art with out stepping on each other toes but in app source control will lock out any changes once the project is checked out and not available until checked back in. If the project is locked out over the Internet it’s not like you can walk down the hall and check the project back in so you land up most times breaking locks which defeats the purpose of having source control in the first place.

So in concept “source control” only works on the local level and best served if you have a large team working on the same project at at the same time

To solve this issue is we simplified our source control procedures using a simple solution.

“Hay Bob don’t mess with such and such project as I’m working on it at the moment.”

Believe it or not working on a small team it just works.

  1. A single source channel does not work period over the Internet. When you combine code with content what is needed by the two groups is totally different as to usability of that group. There are a lot of problems but the big one is the shear amount of data traffic becomes massive the more art content that is added to the mix, aka lfs, and something like working on a single environment can grow to over 100 gigs as the working assets, with out the support of a proper CMS, have to be visible to the project.

Using typical inapp source control would/could mean that the coder would have to pull first, which could take up to 15 minuets, to just do a 30 second push. Adding to the problem that is how your main source control repro grows from a manageable and slick 50 gigs to a bloated 160 gigs. :wink:

So how did we land up solving our issues?

To solve things until something better comes along we broke things down to what is needed rather than trying to make things more convenient by having 3 repos based on the needs of the group.

Content SVN

We created a SVN repo just for art content and environment development that can be pushed and pulled to with out stepping on the toes of the coders.Source control is not even required because art content is art content and the only requirements is to be able to revert the SVN and because of it’s nature there is a ton of backups based on the number of devs mirroring the repo.

Code Git

All of our code stuff is managed through git as this is what git was designed to do. The fact that LFS was hot glued, patched on to the back end, the first place is a statement that the original design did not even account for large project development like video games. In this configuration the coders are happy as it’s business as usual and they can do their thing with out the need for the concerns of the content, big data dumps, and can easily be maintained. GIt lab and bitbuckets are now a usable resource.

Main Build SVN

The main build SVN is were code and content is consolidated to as to what is required by the game. Coders would update the build files and pull what they need from the content SVN and only what is required and use a build bot that will build all of the files into the working game and make it available for play testing. A tip consider adding Steam to the mix early. :slight_smile:

As disconnected as it might sound it works and with out a true definition of what is source control a simple “Hay Bob I’m working on such and such” gets the job done. After all it should be about keeping people working and not about having to steal locks all the time :smiley:

I am still using SVN for everything - over LAN & internet. SVN still serves me well, and even though the files could be big, it is still a no fuss to transfer over internet as internet is pretty fast today. I do not need decentralized commit and centralized server for me simplifies a lot. And of course my team is small (< 5) and therefore, the story may change once we grow…

That’s actually not the case with perforce. Since it works on a per-file basis, you can easily sync just the source folder, resolve conflicts with local changes, and submit your code, without syncing the 3GB of changed art files first.

You can also disable syncing certain folders completely, so as a programmer you can just… not download those 100GB of art source files you’ll never touch at all.

True in a logical kind of way but it’s the coders responsibility to build the changes which also requires art updates to be included as well. It’s a rule as an iteration can mess up what they are working on.

With their own GIT they can do anything they wish.

That and we don’t have the $$$$ for perforce.

Perforce is free for up to 5 users. “Small teams of up to 5 users get free access to Helix Core, the world’s fastest, most scalable versioning engine.”

https://www.perforce.com/how-buy

Yuper but we exceed that limit by 4 times.Subscription based solutions and forced update fees gets expensive real fast

^This! We use this workflow at work for our mobile projects. For our next big thing we use SVN for everything and it works out quite nicely. That repo is about 350gb in total and the checkout is 80gb. Never had problems with it and download speed and checkout speed was fast. Just make sure you use an SSD for better commiting speed.

Yo guys, thanks for your answers.
I guess we have make up our minds and get a mixed solution or get rich :rolleyes:
The steam pipeline is already integrated in our process and we just start our game builds using a slack bot but this git thing is just such a hassle…