Handle few thousands of collisions?

I found out there no noticeable difference if I change the collision responses or not, its enough that I just turn on the collision to something and when trying to move few hundreds of collisions the fps tanks.