Thanks for the suggestions guys !
I’ve never seen that Distance Field AO has a setting there, so that got my hopes up 
Sadly, it didn’t change anything really, which was a bummer.
However, I found another solution for ambient shadows in the very far distance:
I used regular screenspace AO, and set the fadeout distance to a really high amout. Afterwards, I set the actual radius, which is like 150 by default, to 1 or 2 and also put the MIPS to almost 0 as well. This way you still get very sharp AO and it draws shadows underneath every tree, bush, etc. It’s an unusual value to go so low with the radius, but in 50 km distance, the detault radius will just get so large that it’s basically gone - especially with the mips being triggered.
I still can’t make DFAO working in that distance, but at least the regular AO works really good now and nothing is hovering anymore.
Cheers !