Unpredictable Collisions

I would like to simulate raking leaves
(crazy, right?)

Aye, quite!

Have a look at Depenetration setting:

It dictates how quickly intersecting geometries get ejected from one another - low values eject sluggishly. When physic objects are placed in awkward positions where they have nowhere to go, it often causes them to shoot out uncontrollably as the aggressive depenetration setting kicks in.

Start with a value of 10 (or thereabout) and keep increasing it gradually until you find a balance where the leaves no longer get stuck too much into geometry but no longer shoot out like crazy.