I found a typo on one of the #if’s on the tessellation code that was basically ignoring displacement falloff for all tessellated patches
NaniteRasterizer.usf:822
float LowTessDistance = 0.0f;
#if USE_DISPLACEMENT
LowTessDistance = CalcDisplacementLowTessDistance(PrimitiveData, Patches_InstanceData, NaniteView);
#endif
The correct one is USES_DISPLACEMENT
With the incorrect one, this will end up happening:
bDistant[i] = !bOrtho && LowTessDistance > 0.0f && EdgeScale > LowTessDistance;
with 0 bDistant will always be false. And with that the final Tessellation Factor calculation will always have the distance displacement falloff component set to 1.
LowTessMultiple is a hardcoded 0.25 btw:
TessFactors *= NaniteView.LODScale * InvDiceRate * select(bDistant, LowTessMultiple, 1.0f);