Packaging/Shipping build errors from custom C++ plugin/libraries using OpenSSL

Development Programming & Scripting Multiplayer & Networking
, , , , ,
1

Question/Problem

If I build in Development editor configuration, the build is successful. But when trying to package the project, I get lots of already defined in libcurl_a.lib linker errors, because I am linking in libssl.lib and libcrypto.lib via a custom plugin (that relies on custom C++ libraries using OpenSSL). This is a very major issue as it is restricting access to complex libraries built on top of using that OpenSSL version. How can I disable libcurl and related/conflicting modules from builds/packaging?

UATHelper: Packaging (Windows (64-bit)):   libssl.lib(ssl_lib.obj) : error LNK2005: ssl3_undef_enc_method already defined in libcurl_a.lib(ssl_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(ssl_lib.obj) : error LNK2005: SSL_version_str already defined in libcurl_a.lib(ssl_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(t1_lib.obj) : error LNK2005: TLSv1_enc_data already defined in libcurl_a.lib(t1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(t1_lib.obj) : error LNK2005: TLSv1_1_enc_data already defined in libcurl_a.lib(t1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(t1_lib.obj) : error LNK2005: TLSv1_2_enc_data already defined in libcurl_a.lib(t1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(d1_lib.obj) : error LNK2005: DTLSv1_enc_data already defined in libcurl_a.lib(d1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(d1_lib.obj) : error LNK2005: DTLSv1_2_enc_data already defined in libcurl_a.lib(d1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(s3_lib.obj) : error LNK2005: SSLv3_enc_data already defined in libcurl_a.lib(s3_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(cryptlib.obj) : error LNK2005: OPENSSL_NONPIC_relocated already defined in libcurl_a.lib(cryptlib.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(v3_ncons.obj) : error LNK2005: v3_name_constraints already defined in libcurl_a.lib(v3_ncons.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_encrypt already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_decrypt already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ecb_encrypt already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ccm64_encrypt_blocks already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ccm64_decrypt_blocks already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ctr32_encrypt_blocks already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_xts_encrypt already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_xts_decrypt already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_cbc_encrypt already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_set_decrypt_key already defined in libcurl_a.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_set_encrypt_key already defined in libcurl_a.lib(aesni-x86_64.obj)

It is very similar to Shipping plugin with ThirdParty lib with libCURL - Plugins - Epic Developer Community Forums (that was never answered), except I am using another version of OpenSSL instead of another version of libcurl.

Technical details

Unreal Engine: 4.19.2 (pre-built Launcher version).

Compiler/IDE: Visual Studio 2015 Enterprise Update 3

OpenSSL: 1.1.0h

Operating System: Windows 7 Ultimate x64 Service Pack 1

Relevant links studied (irrelevant omitted)

  • https://docs.unrealengine.com/en-us/Programming/UnrealBuildSystem/ModuleFiles
  • https://answers.unrealengine.com/questions/82448/how-to-activate-the-built-in-library-ue4-libcurl-t.html
  • https://answers.unrealengine.com/questions/254591/libcurl-declaration.html
  • https://answers.unrealengine.com/questions/428486/programming-with-third-party-libcurl.html
  • https://answers.unrealengine.com/questions/736377/using-libcurl-withing-a-plugin.html
  • https://answers.unrealengine.com/questions/325380/reading-a-file-online.html
  • https://forums.unrealengine.com/unreal-engine/announcements-and-releases/1374252-unreal-engine-4-18-released

Approaches NOT working

[1] Adding to project DefaultEngine.ini:

[Networking]
UseLibCurl=false

[2] Adding to .Build.cs of both plugin module (linking in the libraries) AND main project:

PublicDependencyModuleNames.Remove("libcurl");
PublicDependencyModuleNames.Remove("HTTP");
PublicDependencyModuleNames.Remove("OpenSSL");
PrivateDependencyModuleNames.Remove("libcurl");
PrivateDependencyModuleNames.Remove("HTTP");
PrivateDependencyModuleNames.Remove("OpenSSL");
PublicAdditionalLibraries.Remove("libcurl_a.lib");
PublicAdditionalLibraries.Remove("libcurl_a");

[3] Unchecking the “Verify Peer” box in Edit -> Project Settings -> Network -> Libcurl.

[4] Disabling all of the TCP/UDP Messaging (and others in the list) plugins under Edit -> Project Settings -> Plugins category/section.

[5] Commenting out libcurl linking in UE_4.19\Engine\Source\ThirdParty\libcurl\libcurl.Build.cs:

// Commented out.
//PublicAdditionalLibraries.Add("libcurl_a.lib");
//PublicDefinitions.Add("CURL_STATICLIB=1");

Produces different errors (seem to be related to OpenSSL more directly this time).

UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(cryptlib.obj) : error LNK2005: OPENSSL_NONPIC_relocated already defined in libeay.lib(cryptlib.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_encrypt already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_decrypt already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ecb_encrypt already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ccm64_encrypt_blocks already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ccm64_decrypt_blocks already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_ctr32_encrypt_blocks already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_xts_encrypt already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_xts_decrypt already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_cbc_encrypt already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_set_decrypt_key already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libcrypto.lib(aesni-x86_64.obj) : error LNK2005: aesni_set_encrypt_key already defined in libeay.lib(aesni-x86_64.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(ssl_lib.obj) : error LNK2005: ssl3_undef_enc_method already defined in ssleay.lib(ssl_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(ssl_lib.obj) : error LNK2005: SSL_version_str already defined in ssleay.lib(ssl_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(t1_lib.obj) : error LNK2005: TLSv1_enc_data already defined in ssleay.lib(t1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(t1_lib.obj) : error LNK2005: TLSv1_1_enc_data already defined in ssleay.lib(t1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(t1_lib.obj) : error LNK2005: TLSv1_2_enc_data already defined in ssleay.lib(t1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(d1_lib.obj) : error LNK2005: DTLSv1_enc_data already defined in ssleay.lib(d1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(d1_lib.obj) : error LNK2005: DTLSv1_2_enc_data already defined in ssleay.lib(d1_lib.obj)
UATHelper: Packaging (Windows (64-bit)):   libssl.lib(s3_lib.obj) : error LNK2005: SSLv3_enc_data already defined in ssleay.lib(s3_lib.obj)

As well as new errors:

UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_easy_unescape
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_free
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_global_init_mem
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_global_cleanup
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_slist_append
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_slist_free_all
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_share_init
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_share_setopt
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_version_info
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_easy_strerror
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_easy_init
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_easy_setopt
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_easy_cleanup
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_easy_getinfo
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_init
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_add_handle
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_remove_handle
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_perform
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_cleanup
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_info_read
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_strerror
UATHelper: Packaging (Windows (64-bit)):   UE4-HTTP.lib(Module.HTTP.cpp.obj) : error LNK2001: unresolved external symbol curl_multi_setopt

[6] Rebuilding Intermediate directory.

[7] Building from Visual Studio in Shipping configuraion (produces identical errors as through Editor packaging).

[8] Disabling most plugins ( SteamVR, OculusVR, CryptoKeys, OnlineSubsystem, OnlineSubsystemUtils, OnlineSubsystemNull, OnlineSubsystemIOS, GoogleCloudMessaging, NetcodeUnitTest, NUTUnrealEngine4, HTML5Networking, UdpMessaging, TcpMessaging, UObjectPlugin, IOSDeviceProfileSelector, AndroidDeviceProfileSelector, LocationServicesBPLibrary, AppleARKit, AndroidPermission, OnlineSubsystemGooglePlay ), most notably AES Network Pakcet Handler and Platform Cryptography plugin (although the latter two aren’t automatically added to the .uproject, they are unchecked in the Editor).

[9] Trying to build as “Modular” instead of default “Monolithic” by adding LinkType = TargetLinkType.Modular; to <GameProjectName>.Target.cs fails with the following errors:

UATHelper: Packaging (Windows (64-bit)):   LINK : fatal error LNK1181: cannot open input file 'C:\Program Files\Epic Games\UE_4.19\Engine\Intermediate\Build\Win64\UE4\Development\UE4-CoreUObject.lib'
UATHelper: Packaging (Windows (64-bit)):   LINK : fatal error LNK1181: cannot open input file 'C:\Program Files\Epic Games\UE_4.19\Engine\Intermediate\Build\Win64\UE4\Development\UE4-Core.lib'

[10] Trying to add the updated 1.1.0 OpenSSL version in the same way the existing 1.0.2 version is already present (in the Engine ThirdParty directory), through C:\Program Files\Epic Games\UE_4.19\Engine\Source\ThirdParty\OpenSSL\OpenSSL.Build.cs. After, adding the module dependency for that PrivateDependencyModuleNames.Add("OpenSSL"); to <GameProject>.Build.cs (instead of linking OpenSSL through the plugin). But that produced the same behavior/errors as manually linking through the plugin.

Test project

I have created a test project as simplified version of the issue. It uses the same setup (custom plugin with custom C++ library using OpenSSL) that simply initializes two OpenSSL structures. There are only a few errors with that test project, but they are of the same nature, so the underlying issue is the same. The project contains pre-compiled libraries so it should be very easy to use.

Small note: Although it’s a .zip archive file, there’s actually a .7z archive inside the .zip archive, due to the 5.2MB upload limit.

2

You can’t disable those because engine needs them, thats why after commenting them out you started to have unresolved symbols errors from HTTP module which use curl.

I think you trying too aggressive methods here just because library in plugin you technicly trying to nuke any use of OpenSSL in rest of the enigne. Insted of destroying one side you should make them cooperate or at least coexist. Is there really API breaking differences between OpenSSL in UE4 and your plugin? did you tried your library in plugin with OpenSSL in UE4? Did you tried upgrading library in UE4 to version that your plugin needs? did you maybe try to modify build scipts in library to rename other version of OpenSSL in plugin?

3

Is there really API breaking
differences between OpenSSL in UE4 and
your plugin? did you tried your
library in plugin with OpenSSL in UE4?

Yes, there are major differences in behavior and protocol support between 1.0.2 and 1.1.0 branches , thus it will not be compatible as the library relies strictly on 1.1.0 behavior.

Did you tried upgrading library in UE4
to version that your plugin needs?

I haven’t tried it, is that possible with the pre-built launcher version? How?

did you maybe try to modify build
scipts in library to rename other
version of OpenSSL in plugin?

I’m sorry, I don’t understand what you are saying. What build scripts in the library, do you mean the build scripts of the plugin linking/exposing the library? If so, then renaming what other version in the plugin, how, why? I’m sorry, it’s unclear what you were trying to say, could you please rephrase that?

4

Probably worth mentioning as a failed attempt [10]

I’ve tried adding the 1.1.0 OpenSSL version in the same way the current UE4 1.0.2 version exists through C:\Program Files\Epic Games\UE_4.19\Engine\Source\ThirdParty\OpenSSL\OpenSSL.Build.cs and added PrivateDependencyModuleNames.Add("OpenSSL"); to .Build.cs (instead of also linking through the plugin). But that produced the same behavior/errors as manually linking through the plugin.

5

Any progress on this? I’m suffering with the exact same problem.

6
7

is anybody found the solution, wanted to use curl inside unreal, but facing this kinda error, 3-4 days trying to find solution, is it engine version issue, in which versions libCurl does work?

8

I got a bit of a backwards solution for this when I was trying to get gRPC and protobuf into UE4: Rather than try to get updated versions of OpenSSL or whatever into Unreal, I opted to build my third party library against the OpenSSL, libeay32, and ssleay32 DLLs from the Unreal Engine. After I did this, I got everything working well. Obviously if you can’t make your own build of whatever third-party library you’re trying to get into Unreal, you’re pretty screwed… but this method sorted me out! Hope it helps.

Rough guide to my steps to build gRPC:

  1. Figure out which DLLs you need (for me it was libeay32.dll and ssleay32.dll)
  2. Copy the required DLLs from the UE4 install to some folder (maybe C:\MyUnrealDLLs)
  3. Build your third party lib (for me, I had to modify my cmake call)

My cmake call:

cmake … -G “Visual Studio 15 2017 Win64” -DgRPC_SSL_PROVIDER=package -DOPENSSL_ROOT_DIR=“C:\OpenSSL” -DOPENSSL_INCLUDE_DIR=“C:\OpenSSL” -DLIB_EAY_LIBRARY_RELEASE=“C:\OpenSSL\libeay32MD.dll” -DSSL_EAY_LIBRARY_RELEASE=“C:\OpenSSL\ssleay32MD.dll” -DOPENSSL_USE_STATIC_LIBS=FALSE

Maybe this helps someone get on the right track?

9

I’ve got the same issue when packaging our project, turns out we need to use the same version of libraries that are used in Unreal Engine source, then the errors are gone, please refer to this link for more details:

10

I was able to find success by adding the following line to my external library Build.cs file:

PublicDependencyModuleNames.Add("libcurl");

and I removed the libraries of my external library that were defining the same symbols from libcurl_a.lib:

PublicAdditionalLibraries.Add(<libs that don't conflict with libcurl>);
11

Library clash has two possible solutions in c++

1- download openssl source code and change exactly 516 variable’s name and re-define all of them. After that you can compile it with perl and ndk. (If this is a dependency for another library, you should change searched variable names in that library, too.)

2- use anonymous namespace

Also today I tried to update OpenSSL’s itself in Unreal Engine source code and actually I could update FEncryptionContextOpenSSL::GenerateKey_RSA() with EVP APIs but Unreal’s pak file system uses RSA* objects in every where and OpenSSL deprecated whole RSA related things. Everything use EVP* PKey right now.

On the other hand, there is a bug about unsafe conversion in 3.0.7 and 1.1.1 family will be deprecated at september 11. If you can wait, Epic stuff will update OpenSSL in the future, especially after that conversion bug be solved.

MBedTLS doesn’t support subject alternative names. So I don’t think they will use that but if they use any other library, you can use OpenSSL without problem.

In short, you can use updated OpenSSL in 2023.

12

@Wooden_Banana how did you solve this ? Any update on your end ?