I think I found the culprit. My floor is set to movable and has vertex paint on it to switch between 2 materials. I found a topic 
Detailed Lighting Buffer plus others not working - Rendering - Epic Developer Community Forums) that mentioned that the lighting has issue with vertex painted meshes for the Detail Light mode. When I set the floor to static (and loose the benefit of my 2 blended material) then it all bake alright. I don’t know why this affect the other objects and not just the floor. But that must be it.