Thank you again !
I can be worth a try with an earlier version of UE to use PhysX indeed. Is it used by default with UE 4.27 (installed from the launcher) ? I read that starting from 4.23 (?), Chaos was introduced and both were usable, but I guess it isn’t as easy as just modifying a value in the project settings right ?
Also, I managed to do a test on another computer, more powerful this time, and I guess that handling 10k/20k Actors in a little space isn’t very conventional too… Neither my CPU or GPU was heavly computing at all, but performances were still poor. I think there’s also a design issue here I need to solve (switching to particles maybe, but I still need to handle each of them one by one and do my collision detecting stuff 
).
Edit : I did a first rough test with UE 4.27, and I get slightly better performances. It’s a good start maybe ^^