Answering in case anyone else will end up with the same issue.
I did not fix it, but I was able to find a work-around by removing the distance field mesh from all of my meshes except the trees (because they needed the extra depth of light to look nicer).
Eventually as I added more stuff to the level the problem reappeared: since I had not deactivated the “create mesh distance fields” any new item had the mesh distance field, so I went to remove it/tick the “affect distance field lighting” in each static mesh and the flickering disappeared.
I eventually moved off lumen, distance field meshes and virtual shadow maps to move onto using cascaded shadow maps as for now it seems much simpler and easier to set up, while also being much less performance heavy.