Was just the first comment that was a bit harsh. Got some great feedback later and some issues that make it might not work well in every scenario. On of them was flickering caused by the scene depth jittering between frames. It was only slightly noticable, but still there. I have greatly reduced this locally and it improved it a lot. 
Here is a breakdown of how it currently works. Some might find it interesting.
First I get the color contrast.
https://imgsli.com/i/49e8389a-6a98-4fd9-b248-67997131faae.jpg
Then I get the depth difference.
https://imgsli.com/i/59793933-f5ab-482a-b7d7-044f041347fd.jpg
Combined I get this mask.
https://imgsli.com/i/6f9ff546-d271-4356-9a08-44924be0d916.jpg
And then I sharpen the image based on this mask. (High value so it’s easier to see)
https://imgsli.com/i/8a788b6e-d6fe-4df0-b2c8-8132f152a74d.jpg
And finally here is a before/after of the scene: Imgsli