I just found a solution for this
Disabling this part of the DLSS plugin “NVIDIA Streamline Core”:
Will get rid of the black eye and freezing problem 
Only having this setting on:
will still give better visual quality and performance in VR than not having DLSS (or FSR) at all