After filling the FDynamicMesh3, calling
UE::Geometry::CopyVertexUVsToOverlay(mesh, *mesh.Attributes()->PrimaryUV());
UE::Geometry::CopyVertexNormalsToOverlay(mesh, *mesh.Attributes()->PrimaryNormals());
Seems to fix it. Not sure why this is needed.
Oh, and a copy of the VertexColors to the VertexColor overlay is also required.