Faces pointing X- in my mesh look strange after building lightmaps

The model is a house I made in Blender and it isn’t yet properly optimized and triangulated, but all other faces are fine.
Like I said in the title, all faces pointing X- look strange, weird, like a different colour in my mesh. I have double checked materials and that doesn’t seem to be the problem.

That’s how it looks, the material is the same as in the other walls. I have made a lightmap UV but have not finished making the material UV, could that be the problem?

Update: Tried removing the SkyLight, ended up with these walls being pitch black.