Dynamic loading is the key 
I have only about 40000 foliage instances in this test. Plus the 1.000.000 trees of course.
I found that collision is the most expensive part about this. Dynamically having collision only on the nearby/needed trees increases the performance drastically.