I think your post gave me enough insight to figure out the issue. I was expecting the VXGI Ambient to work like most instances, where you would use black and it would contribute in the occluded areas. After adjusting the Diffuse Color to something lighter in tone, I realized that its actually contributing like ambient light, not ambient occlusion. Makes sense now. Thanks for that, sometimes its the little things.