OS X cannot find sqlite in ThirdParty

When trying to setup SQLiteSupport on OS X, I have the following files in

[Source Root]/Engine/Source/ThirdParty/sqlite:

35220-capture.png

During build I get the following error:

Compiling with Mac SDK 10.10`

Performing 5 actions (8 in parallel)`

[1/5] Link UE4Editor-SQLiteSupport-Mac-Debug.dylib

ld: library not found for -lThirdParty/sqlite/lib/Mac/Debug/sqlite.a

clang: error: linker command failed with exit code 1 (use -v to see invocation)

I believe the library is in the right place, but the compiler is having issues resolving the file. Any ideas?

Use the full path to the .a file in your *.Build.cs file, not a relative path.

The build file works fine in Windows. It seems odd that I would have to add the library to my build file. That would be going against the pattern originally established by Epic?

Linking rules are simply different on OS X - take a good, hard look at the HLSLCC.Build.cs file for how to correctly link a third-party static library (.a) on OS X. I’ll pass the key elements here:

Public include headers are references by absolute path:

PublicIncludePaths.Add(UEBuildConfiguration.UEThirdPartySourceDirectory + "hlslcc/hlslcc/src/hlslcc_lib");

Then the absolute path to the library folder is found & stored in a temp variable:

string LibPath = UEBuildConfiguration.UEThirdPartySourceDirectory + "hlslcc/hlslcc/lib/"; 

Then on Mac the target library is added an additional library by absolute path:

else if (Target.Platform == UnrealTargetPlatform.Mac)
{
		if (Target.Configuration == UnrealTargetConfiguration.Debug && BuildConfiguration.bDebugBuildsActuallyUseDebugCRT)
		{
			PublicAdditionalLibraries.Add(LibPath + "Mac/libhlslccd.a");
		}
		else
		{
			PublicAdditionalLibraries.Add(LibPath + "Mac/libhlslcc.a");
		}
}

This is the one true way to add third-party static libraries on Mac in UE4.

Thanks for the details. I’ll try this in the build file and see if it can find it. Great start.

Looking at the SQLiteSupport.Build.cs, it seems to be doing this already, but it is still producing the same error when I add SQLiteSupport as a dependency for a plugin.

It clearly isn’t using a full absolute path in the log you posted earlier: ThirdParty/sqlite/lib/Mac/Debug/sqlite.a is a relative path.

The code is very similar to what you have posted:

...
string LibraryPath = "" + UEBuildConfiguration.UEThirdPartySourceDirectory + "sqlite/lib/" + PlatformName + ConfigurationName;
string LibraryFilename = Path.Combine(LibraryPath, "sqlite" + UEBuildPlatform.GetBuildPlatform(Target.Platform).GetBinaryExtension(UEBuildBinaryType.StaticLibrary));

if (!File.Exists(LibraryFilename))
{
	throw new BuildException("Please refer to the Engine/Source/ThirdParty/sqlite/README.txt file prior to enabling this module.");
}

PublicIncludePaths.Add(UEBuildConfiguration.UEThirdPartySourceDirectory + "sqlite/sqlite/");

PublicDependencyModuleNames.AddRange(
	new string[] {
		"Core",
		"DatabaseSupport",
	}
);

// Lib file
PublicLibraryPaths.Add(LibraryPath);
PublicAdditionalLibraries.Add(LibraryFilename);
...

In your post you said that UEBuildConfiguration.UEThirdPartySourceDirectory would help in building the absolute directory? The file is being found during the building of the public library information, but it doesn’t seem to be picked up when linking.

I’m puzzled by this - the path isn’t absolute, which it should be, & I assume that you’ve placed the sqllite folder alongside hlslcc & all its friends in the UE4/Engine/ThirdParty folder?

I’m using the sqlite directory that already exists in github. I just built the library with clang and placed it in what I thought was the appropriate subdirectory (as posted in the question above)

Even when I replace the library with the absolute path on my machine, I get the same error:

Compiling with Mac SDK 10.10

Performing 5 actions (8 in parallel)

[1/5] Link UE4Editor-SQLiteSupport-Mac-Debug.dylib

ld: library not found for -l/Users/afuzzyllama/Development/UnrealEngine/Engine/Source/ThirdParty/sqlite/lib/Mac/Debug/sqlite.a

clang: error: linker command failed with exit code 1 (use -v to see invocation)

Perhaps I didn’t build the library correctly?

If you go into Xcode & go to the messages view, then highlight the ‘Run external build tool’ line and select the button that appears on the right hand side. This will show more details of the build process. See if there are any more details in the log than just that it failed to find the library.

If you’ve built the library for the wrong target (i.e. x86 instead of x86_64) there should be a warning emitted that it is ignoring the library because it is the wrong architecture. That’d be the next most obvious possibility.

Yeah, I already checked under that and the output is exactly the same. I built the library with the following command:

gcc -arch x86_64 shell.c sqlite3.c -lpthread -ldl

Is there anyway to make the UBT invoke the ld with -v without too much effort?

In .cs, in GetLinkArguments_Global add the -v option like the other arguments and see if that emits any more information.

After adding that in, I need to recompile the UBT right?

That should happen automatically as it is a C# application.

Even with adding -v to clang and ld, it still only says file not found. Has someone at Epic successfully linked on OS X to sqlite?

[1/5] Link UE4Editor-SQLiteSupport-Mac-Debug.dylib

Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)

Target: x86_64-apple-darwin14.1.0

Thread model: posix

“/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld”

-demangle

-dynamic

-dylib

-dylib_compatibility_version 4.7.0

-dylib_current_version 0.0.0

-arch x86_64 -dylib_install_name @rpath/UE4Editor-SQLiteSupport-Mac-Debug.dylib

-dead_strip

-headerpad_max_install_names

-macosx_version_min 10.9.0

-syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk

-o /Users/afuzzyllama/Development/UnrealEngine/Engine/Binaries/Mac/UE4Editor-Mac-Debug.app/Contents/MacOS/UE4Editor-SQLiteSupport-Mac-Debug.dylib

-lc++ -rpath @loader_path/

-rpath @executable_path/

-l/Users/afuzzyllama/Development/UnrealEngine/Engine/Source/ThirdParty/sqlite/lib/Mac/Debug/sqlite.a

/System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/Current/MultitouchSupport

/System/Library/PrivateFrameworks/CoreSymbolication.framework/Versions/Current/CoreSymbolication

-framework Cocoa

-framework Carbon

-framework IOKit /Users/afuzzyllama/Development/UnrealEngine/Engine/Intermediate/Build/Mac/UE4Editor/Debug/SQLiteSupport/Module.SQLiteSupport.cpp.o

/Users/afuzzyllama/Development/UnrealEngine/Engine/Binaries/Mac/UE4Editor-Mac-Debug.app/Contents/MacOS/UE4Editor-Core-Mac-Debug.dylib

/Users/afuzzyllama/Development/UnrealEngine/Engine/Binaries/Mac/UE4Editor-Mac-Debug.app/Contents/MacOS/UE4Editor-DatabaseSupport-Mac-Debug.dylib

-lc++

-lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/…/lib/clang/6.0/lib/darwin/libclang_rt.osx.a

ld: library not found for -l/Users/afuzzyllama/Development/UnrealEngine/Engine/Source/ThirdParty/sqlite/lib/Mac/Debug/sqlite.a

The problem is that the Unreal Build Tool is using improper ld parameters. I’ve fixed the issue and submitted a pull request on Github. However, to fix it now you just have to perform two changes.

First, inside Engine/Source/Programs/UnrealBuildTool/Mac/.cs, delete line 751 ( LinkCommand += string.Format(" -l{0}", AdditionalLibrary); ) and paste this code in its place.

EDIT: It’s line 751 in master branch (as seen in the pull request), but it’s line 743 in the version in the release branch.

EDIT 2: If you look at the code submitted to the pull request, make sure you only copy and paste in the code I’ve changed. If you were to copy the entire .cs file from the master branch, to a release branch install you’ll get compile errors due to other unrelated changes to the file.

//Parse out the components of AdditionalLibrary.
//For a common system library, AdditionalLibrary will be just a name. Ex: AdditionalLibrary = "z" is to include "libz.a".
//However, for a custom library, it may be given as a path to the file. Ex: AdditonalLibrary = "ThirdParty/Source/sqlite/lib/Mac/Debug/libsqlite.a"

string libDirectory = Path.GetDirectoryName(AdditionalLibrary);
string libExtension = Path.GetExtension(AdditionalLibrary);
string libName = Path.GetFileNameWithoutExtension(AdditionalLibrary); //expected output from "/ex/lib/dir/mylib.a" is "mylib"

//Note: All of my comments mention ".a" files, however, this also applies to ".dylib" files.
if(libExtension == ".a" || libExtension == ".dylib"){ //If this is a Mac static/dynamic library , given as a path to a specific file. 
	if(!libExtension.StartsWith("lib")){
		Log.TraceWarning("Static library given as path to file named {0}{1}. Continuing, but `ld -l` will actually be searching for files named lib{0}.a or lib{0}.dylib!", libName, libExtension);
	}
	else{
		libName = libName.Substring(3); //Trim off "lib" prefix from "libmylib.a", as the Mac `ld -lmylib` flag will actually search for "libmylib.a". So, if `ld -llibmylib` was given, it'd search for "liblibmylib.a". 
	}
	//Log.TraceInformation("Using new library format for \"{0}\" (lib{0}.a) in search dir \"{1}\"", libName, libDirectory);
	LinkCommand += string.Format(" -L{0}", libDirectory); //Add a new library search path
	LinkCommand += string.Format(" -l{0}", libName); //search for the lib by name. Given the value "mylib" ld will search for "libmylib.a" within the added search directory.
}
else{
	//Log.TraceInformation("Using old library format for \"{0}\" (lib{0}.a)",AdditionalLibrary);
	LinkCommand += string.Format(" -l{0}", AdditionalLibrary); //Leaving this here in the hope that I don't break other functionailty, though accoring to the ld man page, this shouldn't work if AdditionalLibrary is a path to a specific .a file.
}

For the second change, it’s important to note this section from the man ld page:

 -lx         This option tells the linker to search for libx.dylib or
             libx.a in the library search path.  If string x is of the
             form y.o, then that file is searched for in the same places,
             but without prepending `lib' or appending `.a' or `.dylib' to
             the filename.

So, make sure inside ThirdParty/sqlite/lib/Mac/Debug/ and ThirdParty/sqlite/lib/Mac/Release/ you name the sqlite library file libsqlite.a.

You do (edit: I originally thought you didn’t, but you do) need to edit the SQLiteSupport.Build.cs file. You need to change the filename string from “sqlite” to “libsqlite”.

Thank you for the in depth answer. This was the path I was attempting to go down this week, but I didn’t have a chance to finish my investigation. When I’m back on my mac I’ll definitely try this out!