Stereoscopic cubemap that updates with player position

Totally love how we can render out Stereoscopic VRAY from Max/Maya and import them into UE to create a (static) stereoscopic environment.
I know this is far fetched, but bear with me.

I know stereo cubemaps are, by nature, only suitable for single viewpoint applications.
If the player strafes left or right the view has to stay exactly the same (cause only 1 exact viewpoint is included in the stereo-data).

Would it be entirely impossible tho to include the ability to strafe left/right and still get the stereoscopic effect.
I guess we would need to swap out different cubemaps really fast every time the user moves to provide the new ‘perspective’

If the space (the user can navigate around in) is limited tho - would it be viable ?
(I’m developing for the GearVR - so drawcalls might be an issue here)

I know this sounds pretty crazy to pull off or even think about. Still want to put this on the table to maybe get other insights.

I’m asking this to work around the fact that GearVR / mobile dev does not allow for much beauty / post processing. Being able to bring this in from VRay would make it pretty , but again - the texturemaps might become too big to work with.