Hey joelr,
In order to truly understand you methodology with that formula I will need a screenshot of your material and how you are implementing Gerstner’s algorithms. I know that if you don’t set the algorithm up exactly correct then it WILL break your models. Also I have a link to a guy who has already done the math and set the project up in UE4’s node based system.
You can download his project through GitHub. I know the way he is implementing it, it will never be used inside of an engine. I will also link you to our wiki page where a much more cost efficient way implementing waves and translucency has been used.
Link to Gerstner
Link to our wiki
Hope this helps,