Linux (non-Shipping): process aborts at exit with "free(): invalid size" in OPENSSL_cleanup, because UECurl's ConditionalShutdown restores OpenSSL's allocator while OpenSSL still holds memory from Unreal's allocator

General Issues and Bug Reporting (Unreal Engine)
1

Summary

On Linux non-Shipping builds, every process exit ends in a crash after the engine has shut down cleanly. The engine finishes its own exit, main() returns, and then OpenSSL’s exit handler aborts in the C library with “free(): invalid size”.

The process then exits with a crash status (we see 139) instead of the engine’s exit code (for example 143 after a graceful SIGTERM), and the OS writes a core dump on every exit. Gameplay is not affected; exit statuses and crash reporting are.

Engine: 5.8.3 (source build from GitHub, tag commit 396c9f059 “5.8.3 release”)
Platform: Linux x64 (cross-compiled from Windows with the v26_clang-20.1.8-rockylinux8 toolchain), run on Ubuntu 24.04 (glibc 2.39) in a container. Seen in Development and Test builds of a dedicated server; any Linux non-Shipping target that loads the HTTP module should behave the same.
Area: Online / HTTP (libcurl), SSL (OpenSSL 1.1.1t)

Cause (traced live under gdb; the same stack in ten core dumps)

  1. Engine/Source/Runtime/Core/Internal/ThirdParty/UECurl.h:137, UE::Curl::Private::SetCryptoMemoryHooks, called from ConditionalInitialize (from FCurlHttpManager::InitCurl), gives OpenSSL Unreal’s allocator: CryptoMalloc, CryptoRealloc, CryptoFree. Linux non-Shipping only (PLATFORM_UNIX && !UE_BUILD_SHIPPING).
  2. UECurl.h:146, UnsetCryptoMemoryHooks, called from ConditionalShutdown (UECurl.h:207, from FCurlHttpManager::ShutdownCurl at HTTP shutdown), restores OpenSSL’s own allocator, CRYPTO_malloc and friends, which use libc. At that point OpenSSL still holds blocks it allocated through CryptoMalloc.
  3. OpenSSL 1.1.1 lets the restore succeed: while a custom malloc is installed, CRYPTO_malloc never clears allow_customize.
  4. FSslManager::InitializeSsl does not pass OPENSSL_INIT_NO_ATEXIT, so OpenSSL registers its atexit handler. After main() returns, OPENSSL_cleanup frees its remaining blocks with libc free(). In one exit we counted 3,687 such frees of blocks that came from Unreal’s allocator. The first one aborts: glibc finds no valid chunk header.

What Type of Bug are you experiencing?

Online

Steps to Reproduce

  1. Build a project’s Linux dedicated server (or game) in Development.
  2. Run it on Ubuntu 24.04 without the ca-certificates package (for example the plain ubuntu:24.04 container image). The log shows that no certificate bundle was found.
  3. Stop it normally (SIGTERM, or let it exit).

Expected Result

The process exits with the engine’s exit code (143 after a graceful SIGTERM; 0 after a normal quit).

Observed Result

After the engine’s shutdown: “free(): invalid size”, SIGABRT inside OPENSSL_cleanup, a core dump, and a crash exit status.

Affects Versions

5.8

Platform(s)

Linux

For crash reports, include your callstack

#0  PlatformCrashHandler (Signal=6)          Runtime/Core/Private/Unix/UnixPlatformCrashContext.cpp
#5  raise (SIGABRT)
#6  abort
#7  __libc_message_impl
#8  malloc_printerr ("free(): invalid size")
#9  _int_free
#10 free
#11 OPENSSL_cleanup
#12 __run_exit_handlers (status=143)        main() has returned the engine's exit code
#13 exit
#14 __libc_start_call_main

Additional Notes

Why it can hide

SetCryptoMemoryHooks ignores the return value of CRYPTO_set_mem_functions. If anything has already allocated through OpenSSL, the hook is silently refused, and the process then exits cleanly.

On a system with a CA bundle, FUnixPlatformSslCertificateManager::BuildRootCertificateArray loads it (BIO_new_mem_buf) before curl starts. That is the first OpenSSL allocation, so the hook never applies. The fault therefore shows only where no CA bundle is found (“Unable to find a cert bundle” in the log), such as a minimal container image. Installing a debugger package that pulls in ca-certificates makes it disappear.

Fix we apply locally (one line; only Linux non-Shipping changes)

--- a/Engine/Source/Runtime/Core/Internal/ThirdParty/UECurl.h
+++ b/Engine/Source/Runtime/Core/Internal/ThirdParty/UECurl.h
@@ -204,7 +204,7 @@ inline void ConditionalShutdown(FCurlInitializeData& InitializeData)
 	FSslModule& SslModule = FModuleManager::LoadModuleChecked<FSslModule>("SSL");
 	SslModule.GetSslManager().ShutdownSsl();

-	UnsetCryptoMemoryHooks(InitializeData);
+	// OpenSSL keeps Unreal's allocator to the end of the process: it still holds blocks from CryptoMalloc
 #endif // #if WITH_SSL
 }

With this, OpenSSL frees with the allocator that allocated. CryptoMalloc and CryptoFree live in Core, which is never unloaded, so they are still valid when OpenSSL’s exit handler runs. In our tests, ten graceful stops and two refused starts then all ended with the engine’s own exit code, and no core dump was written.

Other fixes Epic may prefer

  • Check the return value of CRYPTO_set_mem_functions, so the hooks apply consistently or not at all. This removes the dependence on the environment, but on its own it does not fix the restore.
  • Initialize OpenSSL with OPENSSL_INIT_NO_ATEXIT and call OPENSSL_cleanup() explicitly before the allocator is restored. This only works if nothing uses OpenSSL after HTTP shutdown.