Directional light's specular highlight shows though when using raytraced distance field shadows

I have my scene set up with distance field shadows, most of my models use a distance field resolution of 3 or 4 and have custom distance field substitution meshes to help it generate a nice clean distance field.

It all works, Omnilights, Ambient Occlusion, Sky Occlusion and all that, but my directional light’s specular shows though even multiple layers of meshes.

You can see in my screenshots, that the specular highlight is visible when in shadowed areas.

Does anyone know what might be the problem?

Make sure you have Reflection Captures near your wall.