Hmm I am not sure what a panorama viewer is. I guess I am confused since normal VR only requires 2 captures.
If you really are computing the whole 360 degrees you will be wasting most of it since a human even in some kind of 360 screen can never see the whole thing at once, unless you look at it as some kinda long-lat chart.
There are tons of papers on the depth thing. Just try looking around. Heres one:
link text