Crash in ISPC::GetPerTrackCompression on AVX and 14th Gen Intel CPU

Development Platform & Builds
, ,
1

Hello,

We are experiencing a crash in ispc::GetPerTrackCompressionPoseScales, GetPerTrackCompressionPoseRotations, and GetPerTrackCompressionPoseTranslations specifically on 14th Gen Intel Processors (i9-14900k).

This crash does NOT occur on AMD processor nor other Intel processors (like Xeons).

The computers with 14th Gen Intel Processors had an upgraded bios with the latest microcode (0x12F)

The crash varies between 15 ~ 120 minutes of soaking.

It does not crash the machine, just our application.

It varies where in the ispc code that it occurs; I took the liberty of un-inlining various functions (DecompressRotation, DecompressTranslation, DecompressScale) and it varied where it would crash. Adding Logging moved the timing to where the crash would no longer occur.

Many times, there is no generated crash dump. That seems alleviated by using newer versions of ISPC, disabling optimizations (-O0), and generating symbols (-g).

I’ve tried disabling optimizations (-O0) and it still crashed.

We are on UE 5.4, and I have tried upgrading ISPC to 1.24.0 (UE 5.6), 1.27.0 and the most recent 1.28.0. None of these fixed the issue.

I have not been able to isolate a particular animation but I do not believe it to be a specific animation at fault (given the nature of the varied time to crash).

The only fix so far has been to disable AVX. Basically limiting our ISPCTargets to SSE4. (So removing AVX, AVX2, and avx512skx-i32x8)

This may be a question to bring up to Intel, but I thought I’d ask here first. Thanks in advance.

2

Steps to Reproduce
Soak test in combat for between 15 to 120 minutes.

3

Hi Patrick

We have seen that problem happening on Fortnite but we haven’t been able to find the source of the problem. We are currently working around the problem by deactivating the use of the ISPC code. In the client’s targetRules , you can define the following to compile the code out:

		// Disabling ISPC for the legacy codecs due to issues with AVX512 target
		GlobalDefinitions.Add("ANIM_CONSTANT_KEY_LERP_ISPC_ENABLED_DEFAULT=0");
		GlobalDefinitions.Add("ANIM_PER_TRACK_COMPRESSION_ISPC_ENABLED_DEFAULT=0");
		GlobalDefinitions.Add("ANIM_VARIABLE_KEY_LERP_ISPC_ENABLED_DEFAULT=0");

Another way to workaround the problem is to switch to the ACL plugin for the compression of the animation: https://dev.epicgames.com/documentation/en\-us/unreal\-engine/animation\-compression\-library\-in\-unreal\-engine

We are actually doing both methods at the same time in Fortnite.

We have seen the problem happen on AMD chips so it doesn’t seem to be specific to a single chip family. It could still be another manifestation of hardware failure caused by overclocking on the Intel CPU. The following article describes the problem and point toward some useful information: Intel Processor Instability Causing Oodle Decompression Failures

Regards,

Martin

4

Thanks for the quick reply.

It turns out we never switched over to using ACL plugin from default. So we will take a look into that.

And I’ll add the defines for disabling ISPC use for legacy anim compression.

~Patrick