I build from source too and this is what I figured. It doesnt need to rebuild the engine along with your project, even for the first build of a new project. Just build the engine in Development - Win64 and Development Editor - Win64 (or any other matching set) once (without your game, from its own solution). Create your project, open it in Visual Studio and switch the startup project to your game. Change Solution Configuration to match the configuration you built the editor with (f.e. Development Editor - Win64). Now take care that when you build your game, that you really do that, Build it, not rebuild it. It won’t compile the engine/editor project then. If you happen to clean the solution or hit rebuild by mistake go take a walk or whatever, because Visual Studio is going to delete and rebuild the editor in the engine source location. Im now psycho about not hitting accidentially anything in UE4 game projects what I used to do often with other projects, that is clean and rebuild. Clean by hand, if you know which files cause trouble with building and need to be cleaned.
Another thing I saw is that stopping a build in Visual Studio doesnt neccessarily mean the build process stops. Even if Visual Studio looks like it did, the Unreal Build Tool and the child cl.exe processes keep running. I dont know if they keep running for their actually sources and then stop or if it keeps build the rest in the background, but I kill them anyway by hand if that happens, as I doubt starting another build on top of it is meaningful or harmless.
If you going to package, the editor or the frontend are going to rebuild the engine (but not the editor) and your code if it sees fit (and I think it almost(?) always does).
My question would be can you exclude the engine source code from the generated solution to be like a project bound to the launcher version of the engine? So that those accidentially rebuild and all the grief and tears dont happen ?