So, a miracle happened:
https://s9.postimg.org/vsj3rigwv/ue4_fixed_lightmaps1.png
full: https://s10.postimg.org/ve3xzevon/ue4_fixed_lightmaps1.png
All I really did was tweaking the material:
https://s9.postimg.org/y0703nlu7/ue4_fixed_lightmaps1a.png
full: https://s9.postimg.org/3te22lj71/ue4_fixed_lightmaps1a.png
Not sure if it makes any sense as I only lerped normal map with flat normal map and swaped order in which they are being lerped.