Two sided material with independent roughness

I have a simple material, single plane geometry, one side has a texture with a glossy finish the other side wants to be a simple matte color. I am having trouble getting the simple color and independent roughness for both sides to work. Here is my set up which kind of works but not really.

Why not just have a different material on each side?