It was a great read, and I hope to get this system as robust as the one in Frostbite :).
So I got the VLS down to 2.2ms for the VLS Render and 0.49 for the VLS Upsample pass. The bilateral filter isn’t in yet so the quality is no were near what it should be but the perf is pretty good right now.
For now if anyone wants better quality then can decrease r.CVarVolumetricLightScatterDownSample to 2 (which makes the VLS Render pass render at 5ms), hopefully soon I can get in the bilateral up-sample filter.