Are there any guidelines for how to maintain and use a custom engine? Specifically for a small team?
Do people build and commit engine binaries to their version control repos?
Or is it better to create a build system that pulls and compiles the latest version of the engine as a part of project setup?
There are a few options that you can use to make a custom Engine version available to your team. The first option is to include the engine source code in your project repo. This may be the simplest option, but it would result in everyone on the team having to build the Engine themselves. It would also significantly increase the size of your repo.
Another option would be to create a separate repo for your Engine source code. This would keep your project and Engine code in different repos, but would still require everyone on the team to build the Engine themselves.
A third option would be to create what is referred to as a Rocket build of your Engine. This is a pre-compiled binary version of the Engine that you can distribute to your team. There is some additional setup required initially, but once it has been built it can be distributed in binary form to other team members so they do not have to build the Engine. It is not something that we support directly at this time, and there is very little documentation for how to do this, but it is something we hope to simplify and expand support for in the future. This forum thread may be helpful if you want to try this option.
There are video tutorials available of how to create an engine from scratch . Maybe the series can be helpful . Just need to learn stuffs like OpenGL ,Visual C++
I am guessing that you are referring to the third option that I mentioned? Unfortunately we do not have much at all with regards to official documentation or examples for this process yet. Hopefully that will improve soon. The information in the Answer on this AnswerHub post may get you started.
I am not familiar with how the Unreal Tournament team builds their version of the Engine, but I would imagine they use a process very similar to the custom build process we use internally when creating the binary version of the Engine that will be released through the Launcher.