Distance To Nearest Surface very costly?

Hi, I recently built a material function that would add dirt on meshes based on distance fields using the Distance to Nearest Surface node.
I added it to my master material that is used for the majority of our assets and added a static switch parameter for turning it on.
It worked great and I applied it to a few instances and made sure it worked in our whole level.

Today, two days later I try opening the main level again and keep getting “Our of video memory” errors despite using a GTX 980TI (6gb vram).
I loaded up a backup and that worked fine, I replaced the maps in the current version of our project with the ones in the backup and that didn’t help.
After that I turned the function off in every material instance, saved, restarted and loaded up the map again and got the same error.
Then I reverted to the older version and only copied the master material and the function over from the newest version to see if it was caused by something else but that didn’t help either.
Even with the function off in every instance and no meshes in the level that have used it I get the error.

Our level is very big and It always uses quite a lot of vram. Streaming pool is set to 1800mb without any errors and with the whole map loaded in editor my VRAM usage is at just below 5gb (yeah I don’t know why it’s so high). With the function in the material but turned off it uses all 6gb and crashes.
I was under the impression that turning it off would ensure it didn’t affect the scene (much) but I’m apparently wrong.

This might be the wrong way to go about it, I couldn’t really find much documentation on using DFAO in materials.
If anyone have some ideas or help to lend I would really appreciate it.