Get compile_commands.json

Development Platform & Builds
, , , , ,
1

Hey, i am using rdm (with emacs) for my game development. rdm requires a compile_commands.json file which is a file that contains all the the compile commands for the code.

I am trying to create one for my game code and for UE4 engine code. usually to generate the compile_commands.json we run “bear” on the compile program, bear expect that the output of the compilation program to be with verbose with all of the command to compile the code but i cant get this from the game compilation.

Is there a way to generate compile_commands.json? Or somehow add --verbose to the clang command?

2

Bear doesn’t depend on the output of the build tool, it depends on being loaded into the build process and intercepting the exec calls to the compiler.

3

The first two steps on Sourcetrail with Unreal Engine 4 - a quick primer suggest a working approach, until native support for JSON Compilation Database is added to the Unreal Build Tool.

Quoting the steps:

  1. Install a patched version of Bear from GitHub - TTimo/Bear at ttimo (note: ttimo branch)
  2. From a clean UE4 tree, capture a compilation database:
    bear -o ./UE4Game.json ./Engine/Build/BatchFiles/Linux/Build.sh UE4Game Linux Development -disableunity -nopch

He points out two issues that currently block Bear from capturing UE4 builds correctly:

4

As of UE 4.24, there is a new mode for UBT: GenerateClangDatabase to do this.

The only documentation I’m aware of is in the 4.24 Release Notes, and there it mentions two new things to UBT: -mode=GenerateClangDatabase to enable compile_commands.json generation and -Filter= to limit the files included.

That will generate a compile_commands.json instead of actually compiling, similar to how the Intellisense code-gen works. I haven’t looked, but I imagine it’s basically the same code-path as used to generate Intellisense.

5

This made my day, I really don’t want to develop using Windows… But I cannot manage to make it work. I haven’t been able to generate, compile_commands.json. I’m not sure if I’m using the UBT tool correctly. Could you share the steps if you have managed to generate the compile_commands.json? Thanks for sharing!

6

Notes from the time give this as the command to run:

C:\unrealengine\Engine\Binaries\DotNET\UnrealBuildTool.exe -mode=GenerateClangDatabase -project="Path\To\My\Project.uproject" -game -engine Target Development Win64

and it will generate C:\unrealengine\compile_commands.json.

That was with a full-source engine from GitHub, it should work similarly with an installed engine.

To be clear, I haven’t done this in a while, this is just going off notes we took at the time we found this.

2 Likes
7

One issue we noted in our testing (UE 4.24.3) is that the compile_commands.json generated doesn’t include the Intermediate directories where the .generated.h files live, so it didn’t actually work for me in the end. I’m not sure if that’s been fixed since.

8

I managed to add generated includes in the compile_commands although you have to modify the UnrealBuildTool itself. I’ve done in on 4.26.0
The file to modify UE_4.26\Engine\Source\Programs\UnrealBuildTool\Modes\GenerateClangDatabase.cs line 123, I’ve added line below

CommandBuilder.AppendFormat(" -I\"{0}\"", Module.GeneratedCodeDirectory);
9

Hi everyone, sorry for waking up an old thread. Ive been trying to get LLVM’s language server Clangd working with Unreal Engine, using this cmd:

& ‘D:\Program Files (x86)\Epic Games\UE_4.26\Engine\Binaries\DotNET\UnrealBuildTool.exe’ Development Win64 BullCowGame “C:\Users\Oliver\Documents\Unreal Projects\BullCowGame-starter-kit\BullCowGame.uproject” -mode=GenerateClangDatabase

I was able to generate a compile_commands.json in my engine directory. However this file doesnt seem to mention all includes necessary. Like CoreMinimal.h for instance is missing and many includes that in fact are mentioned wont be recognized by Clangd anyway.

Do I need to pass some extra arguments or is this simply broken currently? This is the file it generated btw: https://u.teknik.io/sm3mM.json

10

Hi everyone, sorry for waking up an old thread. Ive been trying to get LLVM’s language server Clangd working with Unreal Engine, using this cmd:

& ‘D:\Program Files (x86)\Epic Games\UE_4.26\Engine\Binaries\DotNET\UnrealBuildTool.exe’ Development Win64 BullCowGame “C:\Users\Oliver\Documents\Unreal Projects\BullCowGame-starter-kit\BullCowGame.uproject” -mode=GenerateClangDatabase

I was able to generate a compile_commands.json in my engine directory. However this file doesnt seem to mention all includes necessary. Like CoreMinimal.h for instance is missing and many includes that in fact are mentioned wont be recognized by Clangd anyway.

Do I need to pass some extra arguments or is this simply broken currently? This is the file it generated btw: https://u.teknik.io/sm3mM.json

11

I don’t understand the problem you’re describing. compile_commands.json does not normally list individual header files, unless they’re somehow on the command-line (either compiled, or passed to the compiler as a force-include or similar), so I would not expect CoreMinimal.h to be mentioned in the file.

12

Yep you are probably right, I havent digged in compile_commands.json files before.

I got Clangd to read from the file properly a little after posting this by moving the compile_commands.json from the engine directory to my project directory.

About Kakus comment about adding

 CommandBuilder.AppendFormat(" -I\"{0}\"", Module.GeneratedCodeDirectory);

to GenerateClangDatabase.cs, does this mean that we can also compile our code with Clang even when working on Windows? Would love some guidance on this as I would prefer using Clang over Microsofts cl.exe.

Are there any build flags for UBT or anything like that which lets you build using Clang? And would it be possible to get it working along with Live coding?

13

I’ve managed to compile my project with clang but failed to link it. To do it you have to edit your BuildConfiguration.xml, and add this:

  <WindowsPlatform>
    <Compiler>Clang</Compiler>
  </WindowsPlatform>

You may want to also modify UEBuildWindows.cs in UBT project, line 721

		public static readonly bool bAllowClangLinker = true;
14

I’ve tried with bAllowClangLinker set on and off, but unfortunately without success. Although the issue was with the plugin I use not the project itself. I don’t know whether live coding feature would work with Clang.

15

Hey and thanks for the reply :slight_smile:

You said you didnt get it to link. Was that using the bALlowClangLinker = true; modification?

While I do would prefer Clang Im not sure it would be interesting enough if I would have to deal with longer compile times or having to restart the engine due to it being incompatible with the liver coding feature.

16

With UnrealEngine 5 I was able to do make ARGS="-mode=GenerateClangDatabase" to get it to work.

17

Lucky you, for me running that in the very workspace dir with the Makefile throws with:

make: *** No rule to make target 'CrashReportClient-Linux-Shipping', needed by 'RequiredTools'.  Stop.

I open the file and behold, CrashReportClient-Linux-Shipping is mentioned in RequiredTools: but defined — nowhere.

What now brown cow…

18

I can confirm this works on my side, although I also have some problems (mentioned down below).

Here’s how I make it happen:

  1. Make sure your project is actually built with the build tool. Without it you’ll get warnings due to missing UHT-generated code.
  2. Generate compile_commands.json with Build.sh. In my case my project is called Tlob, so I use TlobEditor module, Linux target (for Windows, it’s Win64), and DebugGame configuration (default would be Development), but you gotta adjust those for you project and directories (just make sure to use absolute directories):
"/home/korn/UnrealEngine/Engine/Build/BatchFiles/Linux/Build.sh" \
	-mode=GenerateClangDatabase \
	-NoExecCodeGenActions \
	-project=/home/korn/Tlob/Tlob.uproject \
	-game \
	-engine \
	TlobEditor \
	Linux \
	DebugGame \
  1. Now, the generated compile_commands.json is located under UnrealEngine’s root directory. You either have to copy or link it into your project’s root directory, but do NOT delete the original file. For example, I do following symbolic link:
ln \
	-sfv \
	~/UnrealEngine/compile_commands.json \
	~/Tlob/compile_commands.json \
;
  1. You may need to restart clangd (or other language server that you use).

Now, the tricky part - problems:

  • Not every clangd version seems to work on my side. I suspect this is because my system-wide headers are clashing with the engine’s headers from its own toolchain. For now I use clangd that comes from mason.nvim package manager, but this is only available for Neovim. Not sure, how someone with Emacs, Vim or VSCode would solve this. Perhaps, you could try sandboxing clangd or passing some additional flags to it.
  • Generation of compile_commands.json is NOT incremental, and takes around 45sec on my machine. To be honest, this is a big issue for me - I add/remove/change one include and I gotta regenerate everything. I’m looking for the solution on this one. (Fixed, see my next comment)
  • Just like someone already mentioned, compile commands doesn’t seem to include Engine’s prelude CoreMinimal.h. I personally don’t like using it, but this is in fact annoying.
  • Indexing takes a lot of time and resources. If you have less than 32gigs of RAM, you may OOM while doing something intensive with the Engine.
  • Seems like this Build tool mode is not well tested and buggy (at least on Linux). I suspect, Epic made it only to support people on MacOSX and we kinda have to deal with it…

Proof it works:

image
image1843×743 93.5 KB

image
image2053×1040 374 KB

image
image880×312 39.7 KB

2 Likes
19

Works brilliantly, thank you!

1 Like
20

Small update on this: