Lumen behaves inconsistently at different times

This is desired Lumen behavious:

This is Lumen if I zoom out or view some other region and then resume camera view after a few minutes - all other settings remaining same.

The problem persists even if I’m not in camera view. So far the desired behaviour have come back randomly sometimes and sometimes not at all. Closing and opening UE5 doesn’t always work either.
Note that in the undesired Lumen behaviour, Lumen scene is still generated and if I increase indirect lighting the shadows will light up - but there is a lot of unnecessary bloom and the richness of colour reflected from far surfaces are lost.