Actually found the problem and the solutions !
It is a problem related to the skylight present in the scene. The skylight seems to affect nanite geometry in a different way than standard meshes. To fix this, I only had to change the skylight type from the default “Stationary” to either "Static or “Movable”.
Link to the original answer and question where I found this : Original Topic