The first of these two images shows the tree without Nanite enabled and the part of the shader that is causing the problem. That part of shader is multiplied with the opacity mask to fade out leaf billboards that are at an angle of roughly 90 degrees to the camera.
The second image shows the problem, which is with Nanite enabled I get these pixelated outlines around parts of the geometry.
I have already confirmed that it is the derivation of the face normal (the world position → ddy X ddx part) in combination with Nanite that is causing the problem. If I use vertex normal instead the problem goes away, but then the shader doesn’t work anymore since the billboards already have smoothed normals.
So my question is, is there a way to get the face normals without it causing problems with nanite. (I guess I could try and remake the tree without edited normals, but it would be better to know if such a way exists.