Encrypted DLL Crashes On Startup

Development Pipeline & Plugins
, ,
1

I help maintain a plugin that makes uses of some helper DLLs. Un-encrypted builds of these DLLs load fine, but the encrypted builds crash the editor immediately at launch on UE 5.8. The same encrypted DLLs worked on 5.7, so this is new 5.8 behaviour. I have tracked this crash to WindowsCallstackTrace.cpp FBacktracer::AddModule(). From what I can tell this is called no matter what when loading a DLL and attempts to construct an unwind table using the function pointers from the DLL. I’m not that familiar with what the encryption does but it seems to jumble these up so they then point to invalid memory which is what causes the crash. Because of the company policy I can’t ship un-encrypted versions of the DLL to make it work. If a check could be added to ensure that the access violation can’t be reached or a way to opt out of the tracing for certain DLLs that would be great. I was able to get around this problem by modifying the engine source to provide a helper function to make sure the data is readable before de-referecing:

static bool BacktraceReadable(const void* Ptr, SIZE_T Size, UPTRINT ModuleBase, UPTRINT ModuleEnd,
	const TCHAR* Name, const TCHAR* Where, uint32 FuncIndex, uint32 NumFunctions)
{
	MEMORY_BASIC_INFORMATION Mbi = {};
	const uint8* Cur = (const uint8*)Ptr;
	const uint8* PtrEnd = Cur + Size;
	bool bReadable = true;
	while (Cur < PtrEnd)
	{
		if (::VirtualQuery(Cur, &Mbi, sizeof(Mbi)) == 0 ||
			Mbi.State != MEM_COMMIT ||
			(Mbi.Protect & (PAGE_NOACCESS | PAGE_GUARD)))
		{
			bReadable = false;
			break;
		}
		Cur = (const uint8*)Mbi.BaseAddress + Mbi.RegionSize;
	}
	if (!bReadable)
	{
		const bool  bInImage = (UPTRINT(Ptr) >= ModuleBase) && (UPTRINT(Ptr) < ModuleEnd);
		const int64 Rva = (int64)(UPTRINT(Ptr) - ModuleBase);
		FPlatformMisc::LowLevelOutputDebugStringf(
			TEXT("[RS-UNWIND] skip module=%s where=%s func=%u/%u ptr=0x%p rva=0x%llx base=0x%p end=0x%p inImage=%d state=0x%08x protect=0x%08x\n"),
			Name ? Name : TEXT("<null>"), Where, FuncIndex, NumFunctions, Ptr, Rva,
			(void*)ModuleBase, (void*)ModuleEnd, bInImage ? 1 : 0, (uint32)Mbi.State, (uint32)Mbi.Protect);
	}
	return bReadable;
}

This just skips the problematic DLL entirely and allows the editor to launch. If it’s possible to add something like this that would be greatly appreciated.

Let me know if I can provide anymore information.

[Attachment Removed]

2

Hi Josh,

I’m guessing this is some sort of obfuscation or runtime decrypted scheme? Since we have no way of knowing if the module is encrypted or not, and I don’t want to add an unconditional VMEM check for each memory access in this code, you can:

  • Make sure the encrypted dll is inside a “ThirdParty” directory so CallstackTrace_FilterModule can filter it out. Ideally this would be configuration-driven, but for most scenarios callstack tracing (when used in detailed memory tracing) is initialized long before the config system is initalized.
  • Another option is to set UE_CALLSTACK_TRACE_USE_UNWIND_TABLES=0 for your target, this disables the unwind table method of doing callstack tracing and uses the slower RtlCaptureStackBackTrace. Since this uses OS functionality callstacks will work.
    [Attachment Removed]
3

Is there another way to opt out of the unwind table method?

No, the unwind method is a compile time switch.

Also with the folder method is that assuming the DLL is apart of the folder structure of the plugin? We pull the DLLs from an special installation folder that’s elsewhere on the machine.

The module is filtered out if the module name contains “ThirdParty”

bool CallstackTrace_FilterModule(FStringView ModuleName)
{
	return !ModuleName.Contains(TEXTVIEW("Binaries")) && ModuleName.Contains(TEXTVIEW("ThirdParty"));
}

So as long as the path contains that string it should be filtered. “ModuleName” here will be the full path to where the dll is loaded from.

[Attachment Removed]

4

I’ll try using a ThirdParty folder. I noticed the define for UE_CALLSTACK_TRACE_USE_UNWIND_TABLES. Unfortunately in my use case I’m only compiling the engine from source for testing purposes. When the plugin is released users will be using the precompiled versions from the launcher. Is there another way to opt out of the unwind table method?

[Attachment Removed]

5

Also with the folder method is that assuming the DLL is apart of the folder structure of the plugin? We pull the DLLs from an special installation folder that’s elsewhere on the machine.

[Attachment Removed]