[outdated, as of this post the latest driver version 528.49 seems to work fine] Speaking of the guy who made it, see his post on Nvidia drivers Epic's GPUlightmass - #423 by Luoshuang
I had major issues baking stuff on 5.1 but after finding that post and reverting to the aforementioned 522.25 all my problems went away.
PS: Also I applied this mentioned edit before reverting to the older driver, it didn’t seem to fix anything for me but technically I applied this edit and then reverted my driver after which everything was perfect Epic's GPUlightmass - #417 by Luoshuang