The issue was indeed due to LOD distribution, I have to use rather extreme settings due to the size of my landscape, as somewhat normal settings lowers my performance from 70fps to 20.
Perhaps I should look into making distant landscapes a background and instead use level streaming for when the player approaches other areas at some point.