I think I found the issue, however I don’t know how to fix it.
I believe that the issue stems from the scene depth capture I am using. It looks like it only captures the Z depth (not sure if that’s the right terminology) and not the ray distance (also not sure of that’s the right terminology). So if you take a scene capture and point it directly at a flat surface, it will output an even value, that’s working its a flat surface, right? Yes-ish but what it fails to do is give the distance based on the focal length, what I would expect of the output of a flat depth capture is a gradient where the center of the capture is the shortest distance and then gradually having a longer distance the further from the center it is.