Trouble with LPV in outdoor lighting

I’m really struggling trying to get LPV to work well. I’m making a procedurally generated game and I’d like to be able to have dark caves without doing anything too hacky, so I’d really like to get dynamic GI working. Currently, the voxels closest to the objects are just way too bright, making it completely useless. Playing around with the limited settings available in the directional light and the post process volume has been of no help.

I can visualize how I want it to look when I first disable LPV. The too-bright source voxels fade away first, leaving just the more diffused bounce light, which looks great. Is there any way to actually make the LPV more diffused?