Download

GitHub dependency zips and latest-preview snapshots

Development Discussion Engine Source & GitHub
#1

I’ve just updated the GitHub README with new instructions for getting UE4 dependencies - there’s no need to manually download and extract zip files for the master or 4.6 preview branch; you can just run the Setup script in a clean repository, and it’ll do everything for you.

As well as downloading the dependencies matching your repository, this will register Git hooks to sync dependencies every time you checkout or merge. It’ll only download the differences needed to update your working tree up to the right state, and it can be safely interrupted or resumed at any point. It won’t clobber changes you make locally without prompting you first. This system works for any commit, not just for the snapshots of ‘latest-preview’ tagged builds that we’ve been limited to until now. If you want to track the master branch in real time, you can! If you want to stick to stable builds verified by our QA team you can track the new promoted branch](https://github.com/EpicGames/UnrealEngine/tree/promoted), which receives merges from master whenever a good build is available.

Of course, this being Epic - the source code for all this is available for you to pore over, and we’ll eagerly take feedback (or pull requests) to improve on it! Some quick pointers - the tool used for downloading is in the repository under Engine/Source/Programs/GitDependencies, and the manifests of files to download are Engine/Build/Commit.gitdeps.xml and Engine/Build/Promoted.gitdeps.xml. Enjoy!

(Note to PS4 and XboxOne developers - you’ll still need to download NDA portions of the Engine from Epic’s FTP site as well as using this system. New zips will be uploaded for every build in the promoted branch.)

#2

This is excellent stuff, thanks!

#3

Smart addition, I just wish this system was implemented directly into the launcher to manage and handle the engine source code and stored remotely on an Epic Games server performing these tasks, I know it would require more networking resources, but working with the GitHUB client hasn’t been the best experience, the GitHUB client keeps crashing on me and it’s been confusing to deal with when issues come up which require troubleshooting it further.

Thank you for this change, it should also hopefully save tons of bandwidth, I went over a 150GB bandwidth limit first time ever awhile back, when you add the source code, plus the regular installs, updates, and marketplace content, with video tutorials and more, oh plus your other third party and standard computer updates, it can quickly hit high on bandwidth usage, I just hope others are aware of that.

#4

setup.bat is ace
well done and thanks Epic :slight_smile:

#5

This is cool.

Wondering if it’s possible for the script to only download the libs for the platform you are on to save on bandwidth. Unfortunately I live in a country where internet isn’t unlimited to 3GB+ each time I want to update libs eats into my allowance easily.

Cheers
Dan

#6

It filters out binaries for Windows, Mac, HTML5 and Android depending on your current platform and which SDKs you have installed. It always gets Linux binaries at the moment due to the possibility of cross-compiling, but I will see if I can fix that.

This new system doesn’t require you to download 3gb every time though - it only downloads deltas from what you currently have, which should reduce your download significantly.

#7

Cool stuff! So do you recommend using the promoted branch? Is the intention to use it the same way is in P4 where we checkout the labels for each release? Using the same branch makes it far better to updated and merge our own forks. I already had some issues when I merged my dev branch with a new release, so fixing it by hand was quite tedious :smiley:

#8

I’ve set up a separate rolling branch for releases (see my post here) - you may want to stick to that; it’s pretty similar to the way the P4 labels work for releases. Promoted builds mainly focus on basic editor functionality, and won’t be anywhere near as refined as the releases.

#9

thats good to know. if we then want to build with new source updates, do we just sync github and re-run dependency script? I have been downoaded source zips from github. so is it just the third party folder that needs copying across?

#10

If you’re using Git, it will automatically download the new dependencies whenever you checkout or merge something. You can run Setup.bat at any time too.

With the source zips it’s a bit trickier to make sure you’ve got a clean sync without downloading everything again - you can’t easily update the source code in place without leaving deleted files around, and there’s no clean way to separate out things in the zip from things in the dependency downloads. There’s the content folder, samples, templates, bits of the ThirdParty folder - all mixed together within the same folders as the source.

#11

Found a very nasty side effect: additional files are not cached anywhere outside the working copy. As a result, after “git clean -fdx” every time quite slowly downloaded 3GB.

#12

Are the dependency files protected to be fetched only by active subscribers, like github access to the sources?

#13

Good point. I’ll try to add caching for the 4.7 release.

No, you can get dependencies for any release you already have.

#14

Looks good. Presumably setup.bat now downloads Required1of2 & Required2of2? Where do I get optional.zip from now for VS2012 support?

#15

No, it doesn’t download anything in zip files any more. All the individual files you need are stored as compressed chunks of data on our servers, and it will only download the files you need. That’ll exclude things like Mac libraries on Windows, or Android files if you don’t have the Android SDK installed.

VS2012 files will be included automatically on Windows.

#16

ZIPs were redistribution friendly .-.

Edit: on second thought that’s a better redistribution method if all the Pc has the same configuration, download it one and zip-it.

#17

A bunch of people have asked for a way to cache downloaded files, so I can certainly look at a way to redirect that caching to a shared location.

#18

ZIPs…

How to include HTML5 dependencies files?

#19

The HTML5 dependencies will be downloaded if you have Emscripten installed (it looks for the EMSCRIPTEN environment variable).

#20

Link to the tutorial showing what options can be passed to Setup.bat :
[TUTORIAL] How to use Setup.bat and how to reduce download size](https://forums.unrealengine.com/showthread.php?54957-TUTORIAL-How-to-use-Setup-bat-and-how-to-reduce-download-size)

Currently using:


Setup.bat -exclude=WinRT -exclude=Linux -exclude=Linux32 -exclude=osx64 -exclude=IOS