Welcome to the Engine Source community!

Hey developers!

We started this forum section to give developers a place to chat about the engine source code itself, creating engine and editor plugins, gossiping about latest engine changes, and of course collaborating with others in the GitHub code community! We love that you guys are hacking on the engine and editor code already.

If you’ve had a chance to associate your account with GitHub and check out the source code on there, we’d love to hear about how the experience is going for you! We are somewhat new to GitHub ourselves at Epic (we use Perforce in the office), but so far we’re really liking the platform.

We’ve seen a great number of people already create their own private forks of the engine code and submit some awesome and interesting changes. We’ve been talking about how we can showcase fun contributions and interesting activity going on in the community source code forks. I’d love to hear your suggestions about that.

And what’s really cool is that developers have already been submitting some great bug fixes and even new tools and engine features as Pull Requests. Thank you to everyone who has submitted fixes back already – we really appreciate it.

I’ve been working on a service that will automatically mirror our developer’s latest engine code changes in Perforce over to GitHub, very soon after the changes were committed here at Epic. This means the master branch in GitHub will be a live stream of the latest commits. I’m really excited about that. So, we’re testing this new service now and it should be live sometime next week if everything goes well. Of course, these live commits will include the latest accepted Pull Request contributions from the community too!

We’re going to keep trying to improve the workflows on GitHub. For example, we know the situation with the required dependency downloads are not really ideal. We’ve heard some excellent suggestions already about different ways we could potentially handle that, including on-demand downloading of dependencies and using something like Git Annex to host the larger files. We’ll continue to ponder about this to find a better approach, but we also are trying to keep things as clear and simple as possible.

We hope you’re enjoying working with the engine code on GitHub so far. Keep the feedback coming.

Nicely done Mike. I got the source a little while back, and also have a few ideas as to what part needs some improvement, but haven’t had time to dive into the huge code base of the engine/editor. Hopefully by the time I’m there the live changes are already available.

Thanks for the great job you all.

1 Like

This is super exciting. I’m jazzed to actually learn my way around the engine and contribute something useful. What do you guys think about opening up the issues section on Github? It might preempt some incomplete pull requests in the future and also give community developers some good starting points.

Thanks again Mike, what you guys are doing is really great.

I have been thinking about it, but I’m not sure how to make that work yet. We want to explore somehow exposing our current issue-tracking system to you guys instead of creating a whole other one off to the side. We’re worried that if we turn on the GitHub issue tracker we won’t be able to give it proper attention. But, I do like how nicely it integrates with the other features on GitHub. The current outlet for reporting bugs is to use AnswerHub – we watch it pretty closely and a lot of issues get logged from there. But we know we need to make roadmaps, ongoing efforts and our task backlogs exposed to the community more readily. We’re working on it!!

Hey Mike,

All of this sounds great. I’m really looking forward to having the master branch be a live stream of the changes going into your perforce repo.

One thing I’m curious to hear / debate about is the process Epic is using to pull in changes submitted on GitHub. I have some concerns, so maybe shedding some light on the process would help. When accepting a pull request, is care taken to reformat the pull request when it goes into the internal perforce repo, eg. fixing code style issues (naming conventions, tabs/spaces, etc) and other issues (use of stl headers / library, etc)?

I would really encourage having a process here, either Epic internally fixes these once a pull request is accepted or have the author fix these issues before the pull request gets accepted. I would also encourage spelling these guidelines out to the community. I know this seems a bit bike shedding but it is important to be consistent, especially when you have lots of contributions.

Another concern I have is with the binary required files. My concern is what happens when one or more of these binaries are updated or even removed from the zips and how we can keep our locally extracted binaries in sync. Perhaps we need some kind of batch/shell scripts to clean out the binaries in preparation for extracting new ones to ensure outdated / removed binaries are removed locally? There also needs to be clearer notification of when downloading these required files is necessary between engine updates.

Hey xgalaxy. Thanks for posting. Very good questions!

We will work on communicating the process for code submissions. It will be an ever-evolving process I’m sure. Our engine team has many years of experience with dealing with community code submissions, as many licensees have contributed awesome features and improvements over many years. We try to maintain a high standard for changes to get merged into our mainline. One of my goals over the next few months is to try to articulate what our expectations are, to help improve the odds of submissions being accepted without causing much rework. We have an abundance of incarnate dynamic knowledge that we’re excited to figure out how to share without writing piles and piles of static paragraphs. My goal is to get the various “vision holders” out in the open to discuss what our plans are, in whatever format works for them. The idea is that you guys should have access to the same information that I have, even if it came from a hallway chat.

As for the binaries, we’re aware and are working on that problem. It is not an elegant merge right now, and there is no easy programmatic way to reconcile the new version’s dependency binaries with the last yet. At Epic we use Perforce to store all binaries and dependencies, but with GitHub it was not reasonable to do that. Ideally you guys aren’t having to change dependency binaries much for your own purposes. I tried to choose files for the auxiliary zips that would not cause developers problems if they do not clean them out with every release. However, we will solve it as soon as we can.

One idea we’re looking at is to automatically handle reconciling dependencies automatically by a program that runs before project files are generated. It would download the “right stuff” on demand and you wouldn’t have to worry about the reconcile unless there were local conflicts. Another idea we’re looking at is using the Launcher to help with the GitHub extra dependencies for your newly-sunk version of the GitHub code.

It will probably be a while before we have a super-elegant solution, so please bare with is. We want to be careful not to over-engineer this. If we could just put everything in GitHub we would probably do that. :slight_smile: Some folks suggest using Git Annex, but it is critical to us to keep this as simple and straightforward as possible. We don’t want to force people to use the DOS/terminal box in day 0.

Thanks for creating this new section Mike!

:slight_smile:

Rama

Hi Mike!

Let me introduce myself, and tell you how we handled a similar situation in my company; sources with full history weight a hundred of MB, while binaries dependencies with full history across many years was around 1.5GB)

Where I work, I have introduced Git to replace SVN, setup an appropriate workflow and documented it, as long as the chosen tool (SmartGit by the way, easy and powerful GUI); we have around a hundred developers, working on small teams ranging from 1 or 2 to 20+ devs.

To handle binary dependencies, we started by just letting some of them inside (like released build) and using git submodules for frameworks, SDKs and such.
That worked nicely, and was efficient despite what is said against Git: with 1.4 GB it was more space efficient than SVN! And the cloning took not even twice as long as a first SVN checkout (16 min vs 12 min).

But after a while, we decided to optimize further by more strictly separating binary dependencies from core sources and tools; main repository still contains the build system and its runtime (Bakefile and Python), but any heavy dependency is now out, all in submodules. Thus the main repo now weights around 2OOMB, which is more appropriate for task like branch switching).

=> So, the point is that to optimize Git usage you can separate your project in a few coherent parts, following their different lifecycles, and then submodules are really appropriate.

Hope that help…
Cheers!

edit: typo

Git submodules

I should have said a few more things about git submodules;

  1. Submodules are standard: no extension, contrary to git-annex
  2. Submodules are quite easy to use and understand for the average developer: that is, until you start to modify/commit/mess(!) with them (see below)
  3. Submodules should be use to handle part of the projects that you don’t expect the community to modify often (see above): already the case with sources + 2 ZIP files
  4. Submodules should be use to handle part of the projects with different lifecycles: ThirdParty vs source code
  5. Submodules can be cleaned/recreated if too cluttered**: I’ve used git BFG (but not required before a long time)

I think you already have some quite clear organisation in the ZIP:

  • Engine/ThirdParty (source & binaries)
  • Templates
  • Samples/StarterContent
  • Extras
  • Content
  • Documentation

What do you think?

Hey, UE4 source code received a bunch of updates on GitHub! Seems like live updates are coming…

So excited!

Edit: @Mike it definitely looks like you are testing your “service that will automatically mirror developer’s latest engine code changes in Perforce over to GitHub” :slight_smile:

Yes, we are running a script to get it caught up with the backlog of commits from the last few weeks. I figured that would be a great way to test everything and to give you guys a preview of how it will all look. :slight_smile: We are still getting everything in order, so please bare with us! Even though GitHub will soon have the very latest code changes, I won’t have the process for uploading required dependency snapshots for a day or more, until which the latest code probably won’t compile. But yes, we are super excited too! :slight_smile:

Nice to hear, I can wait a few days, I will give it a try this weekend!

I’m currently looking at commits from 30 minutes ago on upstream/master. So awesome. Thanks again!

Hi. I am looking to make a small alteration to the UDK editor to allow for material types to be input on models (which would then be set up by an ini file with various stats on each material, like how much it can bend / stretch / distort etc. before breaking), i.e malleable metal, cast / brittle metals, wood, thermo plastic, thermo-setting plastic etc etc. If you could give me an idea of how easy this would be and where in the code I would begin to make the alteration, that would be greatly appreciated. Also, can you tell where I download the source code from. Thanks a lot. Kind regards, .

Why is the unreal staff so unresponsive?
I have posted few questions and no one has even bother to say you can’t do that.

I see, I guess the answer is not.
I am cancelling my subscription then. I try my luck with some other engine them.

Thank you Unreal team.

Before you pick up your ball and go home… you do realize that it’s three days before GDC, the biggest and most important event in our industry, right? An event in which a good portion of the Epic employees participate? An event that Epic is a principle sponsor of?

Maybe expecting someone from Epic to personally answer your questions and to respond within 24 hours isn’t exactly reasonable, well… ever, but especially given that a good percentage of the company is either in San Francisco or en route to get set up for the most important event of the year. Hell, my bet is that most of the company is going through major sleep deprivation right now.

Just a thought.

I has not being 24 hour, not even a week. I has being more than a moth.
I am a very patient person.
I have post few legitimate question that I have not found answer in the wiki and not where else.
The engine source code, is not a trivial talk to navigate just to add that Path file to a library.
This is no a Makefile, or CMake, or Jam, or any documented built tool, they created this own method and if Epic decided to invert their own Build tool, then they most explain how it to be used by the end user.

This is my second month on this, I will wait until after the GDC thing.

, I think you’re looking for AnswerHub.

Although we participate in a lot of discussions here, this forum is largely community driven and meant for discussion between UE4 users. On AnswerHub, all questions are monitored by us, which makes it the preferred destination for support requests like yours.

I am really excited to be there with your genius, thank you.