One of the problems that you will have is that most head-tracked VR systems don’t have the same camera location for every orientation, they have an offset on their location that’s affected by the rotation to simulate the way human heads work (rotating from the neck with eyes in front of that by a few inches), so you’re going to need much more data than what would be the equivalent of 2 cubemaps. You essentially need 2 new cubemaps for every possible pitch change if you want passable head tracking, and then if you want to do roll too you need even more, and if you want to be tracking translation even more.
The common solution is to not try to pre-render because you’ll either have so much data it won’t be worth it, or you’ll have to be doing complex image transforms at runtime if you want it to feel right. If you have 3d data already, it’s easier to just render it realtime, which you can do trivially.