Dealing with Allocator Mismatches with External Libraries

Dealing with Allocator Mismatches when using external Libraries

Unreal Engine uses its own allocators, a.k.a. new()/delete() implementation, such as mallocbinned2. Deleting memory that has been allocated outside of Unreal (e.g. from a library/DLL that uses standard new/delete implementations) through Unreal’s allocator will usually result in crashes. The same is true for the opposite, when memory that comes from Unreal’s allocator is being passed to a library that uses the standard allocator to free it.
This issue can often manifest itself when using standard library classes like std::string or passing these through DLL boundaries, e.g. a library function that returns a std::string object.

Example Error messages:

FMallocBinned2 Attempt to realloc an unrecognized block 000002025F080000 canary == 0x0 != 0xe3
malloc: *** error for object 0x3140000009c5b: pointer being free was not allocated

Usually the call stack will contain some library code and then go through one of the FMemory::Malloc()/Free()/Realloc() calls of the Engine API.

To validate whether this is the case the engine can be started with the “-ansimalloc” flag. This will disable all custom allocators and use the new/delete functions of the standard library.

How to deal with Allocator Mismatches

There are a few possible solutions/workarounds for dealing with allocator mismatches.

Libraries with Custom Allocator support

Some libraries have an API to provide custom allocator functions, making it possible to supply a user-defined function for new()/delete() or Malloc()/Free() calls.

These can be used to let a library use Unreal’s memory functions, which can be found in the FMemory class in UnrealMemory.h:

FMemory::Malloc()
FMemory::Realloc()
FMemory::Free()

Since Unreal Engine 5.3 each module will also have the following local functions defined that can be used instead of referencing FMemory directly, see ModuleBoilerPlate.h:

void* StdMalloc( size_t Size, size_t Alignment );
void* StdRealloc( void* Original, size_t Size, size_t Alignment );
void StdFree( void *Ptr );

Static Linking

Unreal does overload C++'s new() and delete() operators to forward allocations to the FMemory API.
This only works for code compiled through Unreal’s build environment.

If you are using a dynamic library (.dll) and you have the option of linking this library statically instead (e.g. only a .lib) then the linker should replace the libraries allocations with Unreal’s new()/delete() hooks.

Using the default/standard Allocator

Starting Unreal with the -ansimalloc parameter can be used as a last resort if no other workarounds are applicable. This is not recommended, since Unreal’s build-in allocators usually have better performance at runtime.

I could see tooling this. At least for Windows.
First of all for all your modules #undefine malloc, then #define malloc FMemory::Malloc(args) kind of thing. And at the top header redirect new and delete to the allocators you list.

Make it a practice to not use standard names, use the FMemory ones across the board.
Finally make a post build Python processing tool using LIEF or pefile module to parse the IAT of all built executable modules to capture any and all uses of CRT malloc, new, delete, etc.
Build failure is the use of any of these problem APIs.
Nip the problem in the bud before it manifests in crashes…

I am trying to build a library that can be used by unreal or other systems but I heavily rely on STL and build my components as a set of dlls for various reasons. How do I support allowing unreal developers to provide a custom allocator(Unreals allocator)?