And as usually, I found a solution minutes after writing the post
If you ever run into this kind of issue, make sure you never blend multiple VertexInterpolators, but always use just one after the blending:
