The problem with VR is that it’s technology is premature. The headsets are not comfortable to use, are extremely expensive, and have horrible image quality. In between this and the mediocre adoption rate which also means fewer good games for it and we have the current state of the VR industry.
Then there is also the motion sickness. This is the single biggest reason for why VR might simply never take off even if the technology improves.