I remade the door a little bit ticker, changed and played with two sided material and double sided geometry, but nothing changed. The skylight settings are pretty simple and the world settings just have some small adjustments (except on the Volumetric Lightmap). The floor is not big, and I increased the lightmap just for the sake of testing, but not even with 1024 gave me good results.
The baked door as static looks fine, but I want to use as a movable object. Maybe the volumetric lightmap still has a looong way to go before being usable for archviz?