Ok, thanks! One problem with this is that light bake gives me text that tells me to decrease lightmap resolution caused by that large building. So somehow I need to optimize it even more.
I think I got it working. I made uv maps better, more square and decreased lightmap quality to 16, it looks suprisingly good even with that low resolution.