This is likely an issue on the VR runtime side. At least the SteamVR OpenXR runtime will not clear stereo layers if you don’t render a frame after turning the layer off. This might violate the OpenXR spec and is likely not intended behavior.
More info here: