Nvidia VXGI Specular Performance Issue

I there, I am using Unreal 4.18.3 with Nvidia VXGI. It looks great so far and runs very well. I do have one issue regarding my materials. I’m reworking my landscape materials to be more performance friendly. It is 4 layers, each with textures for diffuse, specular and normal. I also am using pixel depth to blend between large and small versions based on distance for close up detail. With VXGI specular enabled, my FPS drops to between 10 and 20 on my GTX 1080ti. When I disable VXGI specular tracing, it shoots up to 170 -180 as it should be.

Are there certain settings in my materials that I need to be careful of when using the specular tracing feature?

I realize that this is more regarding the use of VXGI, but I’m checking here as I know that at least a few of you use it.