I feel stupid…
The problem wasn’t LOD or Light bake, the problem was InstancedStaticMesh.
Static Mesh has lightmap for each LOD, but Instanced Static Mesh only uses LOD0 lightmap and applies it to each LOD down.
I had to re-make lightmaps for each LOD, so that they are layered on top of each other perfectly.