Announcement

Collapse
No announcement yet.

Welcome to the Engine Source community!

Collapse
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Welcome to the Engine Source community!

    Hey developers!

    We started this forum section to give developers a place to chat about the engine source code itself, creating engine and editor plugins, gossiping about latest engine changes, and of course collaborating with others in the GitHub code community! We love that you guys are hacking on the engine and editor code already.

    If you've had a chance to associate your account with GitHub and check out the source code on there, we'd love to hear about how the experience is going for you! We are somewhat new to GitHub ourselves at Epic (we use Perforce in the office), but so far we're really liking the platform.

    We've seen a great number of people already create their own private forks of the engine code and submit some awesome and interesting changes. We've been talking about how we can showcase fun contributions and interesting activity going on in the community source code forks. I'd love to hear your suggestions about that.

    And what's really cool is that developers have already been submitting some great bug fixes and even new tools and engine features as Pull Requests. Thank you to everyone who has submitted fixes back already -- we really appreciate it.

    I've been working on a service that will automatically mirror our developer's latest engine code changes in Perforce over to GitHub, very soon after the changes were committed here at Epic. This means the master branch in GitHub will be a live stream of the latest commits. I'm really excited about that. So, we're testing this new service now and it should be live sometime next week if everything goes well. Of course, these live commits will include the latest accepted Pull Request contributions from the community too!

    We're going to keep trying to improve the workflows on GitHub. For example, we know the situation with the required dependency downloads are not really ideal. We've heard some excellent suggestions already about different ways we could potentially handle that, including on-demand downloading of dependencies and using something like Git Annex to host the larger files. We'll continue to ponder about this to find a better approach, but we also are trying to keep things as clear and simple as possible.

    We hope you're enjoying working with the engine code on GitHub so far. Keep the feedback coming.


    --Mike

    #2
    Nicely done Mike. I got the source a little while back, and also have a few ideas as to what part needs some improvement, but haven't had time to dive into the huge code base of the engine/editor. Hopefully by the time I'm there the live changes are already available.

    Thanks for the great job you all.

    Comment


      #3
      Originally posted by Mike Fricker View Post
      I've been working on a service that will automatically mirror our developer's latest engine code changes in Perforce over to GitHub, very soon after the changes were committed here at Epic. This means the master branch in GitHub will be a live stream of the latest commits. I'm really excited about that. So, we're testing this new service now and it should be live sometime next week if everything goes well. Of course, these live commits will include the latest accepted Pull Request contributions from the community too!
      This is super exciting. I'm jazzed to actually learn my way around the engine and contribute something useful. What do you guys think about opening up the issues section on Github? It might preempt some incomplete pull requests in the future and also give community developers some good starting points.

      Thanks again Mike, what you guys are doing is really great.
      Work in Progress: King of Kalimpong
      piinecone.com | youtube | @p11necone

      Comment


        #4
        Originally posted by piinecone View Post
        What do you guys think about opening up the issues section on Github?
        I have been thinking about it, but I'm not sure how to make that work yet. We want to explore somehow exposing our current issue-tracking system to you guys instead of creating a whole other one off to the side. We're worried that if we turn on the GitHub issue tracker we won't be able to give it proper attention. But, I do like how nicely it integrates with the other features on GitHub. The current outlet for reporting bugs is to use AnswerHub -- we watch it pretty closely and a lot of issues get logged from there. But we know we need to make roadmaps, ongoing efforts and our task backlogs exposed to the community more readily. We're working on it!!

        --Mike

        Comment


          #5
          Hey Mike,

          All of this sounds great. I'm really looking forward to having the master branch be a live stream of the changes going into your perforce repo.

          One thing I'm curious to hear / debate about is the process Epic is using to pull in changes submitted on GitHub. I have some concerns, so maybe shedding some light on the process would help. When accepting a pull request, is care taken to reformat the pull request when it goes into the internal perforce repo, eg. fixing code style issues (naming conventions, tabs/spaces, etc) and other issues (use of stl headers / library, etc)?

          I would really encourage having a process here, either Epic internally fixes these once a pull request is accepted or have the author fix these issues before the pull request gets accepted. I would also encourage spelling these guidelines out to the community. I know this seems a bit bike shedding but it is important to be consistent, especially when you have lots of contributions.

          Another concern I have is with the binary required files. My concern is what happens when one or more of these binaries are updated or even removed from the zips and how we can keep our locally extracted binaries in sync. Perhaps we need some kind of batch/shell scripts to clean out the binaries in preparation for extracting new ones to ensure outdated / removed binaries are removed locally? There also needs to be clearer notification of when downloading these required files is necessary between engine updates.
          Last edited by xgalaxy; 04-19-2014, 02:36 AM.

          Comment


            #6
            Hey xgalaxy. Thanks for posting. Very good questions!

            We will work on communicating the process for code submissions. It will be an ever-evolving process I'm sure. Our engine team has many years of experience with dealing with community code submissions, as many licensees have contributed awesome features and improvements over many years. We try to maintain a high standard for changes to get merged into our mainline. One of my goals over the next few months is to try to articulate what our expectations are, to help improve the odds of submissions being accepted without causing much rework. We have an abundance of incarnate dynamic knowledge that we're excited to figure out how to share without writing piles and piles of static paragraphs. My goal is to get the various "vision holders" out in the open to discuss what our plans are, in whatever format works for them. The idea is that you guys should have access to the same information that I have, even if it came from a hallway chat.

            As for the binaries, we're aware and are working on that problem. It is not an elegant merge right now, and there is no easy programmatic way to reconcile the new version's dependency binaries with the last yet. At Epic we use Perforce to store all binaries and dependencies, but with GitHub it was not reasonable to do that. Ideally you guys aren't having to change dependency binaries much for your own purposes. I tried to choose files for the auxiliary zips that would not cause developers problems if they do not clean them out with every release. However, we will solve it as soon as we can.

            One idea we're looking at is to automatically handle reconciling dependencies automatically by a program that runs before project files are generated. It would download the "right stuff" on demand and you wouldn't have to worry about the reconcile unless there were local conflicts. Another idea we're looking at is using the Launcher to help with the GitHub extra dependencies for your newly-sunk version of the GitHub code.

            It will probably be a while before we have a super-elegant solution, so please bare with is. We want to be careful not to over-engineer this. If we could just put everything in GitHub we would probably do that. Some folks suggest using Git Annex, but it is critical to us to keep this as simple and straightforward as possible. We don't want to force people to use the DOS/terminal box in day 0.

            --Mike

            Comment


              #7
              Thanks for creating this new section Mike!



              Rama
              UE4 Marketplace: Melee Weapon Plugin & Compressed Binary Save System Plugin | Rama's C++ AI Jumping Videos | Vertex Snap Editor Plugin

              ♥ Rama

              Comment


                #8
                Originally posted by Mike Fricker View Post
                As for the binaries, we're aware and are working on that problem. [...] If we could just put everything in GitHub we would probably do that.
                Hi Mike!

                Let me introduce myself, and tell you how we handled a similar situation in my company; sources with full history weight a hundred of MB, while binaries dependencies with full history across many years was around 1.5GB)

                Where I work, I have introduced Git to replace SVN, setup an appropriate workflow and documented it, as long as the chosen tool (SmartGit by the way, easy and powerful GUI); we have around a hundred developers, working on small teams ranging from 1 or 2 to 20+ devs.

                To handle binary dependencies, we started by just letting some of them inside (like released build) and using git submodules for frameworks, SDKs and such.
                That worked nicely, and was efficient despite what is said against Git: with 1.4 GB it was more space efficient than SVN! And the cloning took not even twice as long as a first SVN checkout (16 min vs 12 min).

                But after a while, we decided to optimize further by more strictly separating binary dependencies from core sources and tools; main repository still contains the build system and its runtime (Bakefile and Python), but any heavy dependency is now out, all in submodules. Thus the main repo now weights around 2OOMB, which is more appropriate for task like branch switching).

                => So, the point is that to optimize Git usage you can separate your project in a few coherent parts, following their different lifecycles, and then submodules are really appropriate.

                Hope that help...
                Cheers!

                edit: typo
                Last edited by SRombauts; 04-20-2014, 09:49 AM.
                [Gamedev programmer at Darewise (Paris) - We are hiring]
                UE4 Git LFS 2.x Source Control Plugin 2.14-beta for UE4.23 - (v1 integrated by default since UE4.7)
                UE4 Plastic SCM Source Control Plugin 1.4.6 for UE4.23 - (integrated by default in Beta status since UE4.24)

                Comment


                  #9
                  Git submodules

                  I should have said a few more things about git submodules;
                  1. Submodules are standard: no extension, contrary to git-annex
                  2. Submodules are quite easy to use and understand for the average developer: that is, until you start to modify/commit/mess(!) with them (see below)
                  3. Submodules should be use to handle part of the projects that you don't expect the community to modify often (see above): already the case with sources + 2 ZIP files
                  4. Submodules should be use to handle part of the projects with different lifecycles: ThirdParty vs source code
                  5. Submodules can be cleaned/recreated if too cluttered[/B]: I've used git BFG (but not required before a long time)


                  I think you already have some quite clear organisation in the ZIP:
                  • Engine/ThirdParty (source & binaries)
                  • Templates
                  • Samples/StarterContent
                  • Extras
                  • Content
                  • Documentation


                  What do you think?
                  [Gamedev programmer at Darewise (Paris) - We are hiring]
                  UE4 Git LFS 2.x Source Control Plugin 2.14-beta for UE4.23 - (v1 integrated by default since UE4.7)
                  UE4 Plastic SCM Source Control Plugin 1.4.6 for UE4.23 - (integrated by default in Beta status since UE4.24)

                  Comment


                    #10
                    Hey, UE4 source code received a bunch of updates on GitHub! Seems like live updates are coming...

                    So excited!

                    Edit: @Mike it definitely looks like you are testing your "service that will automatically mirror developer's latest engine code changes in Perforce over to GitHub"
                    Last edited by SRombauts; 04-23-2014, 06:20 PM.
                    [Gamedev programmer at Darewise (Paris) - We are hiring]
                    UE4 Git LFS 2.x Source Control Plugin 2.14-beta for UE4.23 - (v1 integrated by default since UE4.7)
                    UE4 Plastic SCM Source Control Plugin 1.4.6 for UE4.23 - (integrated by default in Beta status since UE4.24)

                    Comment


                      #11
                      Originally posted by SRombauts View Post
                      @Mike it definitely looks like you are testing your "service that will automatically mirror developer's latest engine code changes in Perforce over to GitHub"
                      Yes, we are running a script to get it caught up with the backlog of commits from the last few weeks. I figured that would be a great way to test everything and to give you guys a preview of how it will all look. We are still getting everything in order, so please bare with us! Even though GitHub will soon have the very latest code changes, I won't have the process for uploading required dependency snapshots for a day or more, until which the latest code probably won't compile. But yes, we are super excited too!

                      --Mike

                      Comment


                        #12
                        Nice to hear, I can wait a few days, I will give it a try this weekend!
                        [Gamedev programmer at Darewise (Paris) - We are hiring]
                        UE4 Git LFS 2.x Source Control Plugin 2.14-beta for UE4.23 - (v1 integrated by default since UE4.7)
                        UE4 Plastic SCM Source Control Plugin 1.4.6 for UE4.23 - (integrated by default in Beta status since UE4.24)

                        Comment


                          #13
                          Originally posted by Mike Fricker View Post
                          This means the master branch in GitHub will be a live stream of the latest commits.
                          I'm currently looking at commits from 30 minutes ago on upstream/master. So awesome. Thanks again!
                          Work in Progress: King of Kalimpong
                          piinecone.com | youtube | @p11necone

                          Comment


                            #14
                            Hi. I am looking to make a small alteration to the UDK editor to allow for material types to be input on models (which would then be set up by an ini file with various stats on each material, like how much it can bend / stretch / distort etc. before breaking), i.e malleable metal, cast / brittle metals, wood, thermo plastic, thermo-setting plastic etc etc. If you could give me an idea of how easy this would be and where in the code I would begin to make the alteration, that would be greatly appreciated. Also, can you tell where I download the source code from. Thanks a lot. Kind regards, Pete.

                            Comment


                              #15
                              Why is the unreal staff so unresponsive?
                              I have posted few questions and no one has even bother to say you can't do that.

                              Comment

                              Working...
                              X