If someone is interested I just saw this video about what we are trying to achieve here but in Substance Designer.
He basically take the world space vertex color, blurs the result and convert it in tangent space. I wonder if this is possible in UE4 or if what I just wrote makes any sense…