Physics performance question


I started playing around with physics this morning and I m curious whether the physics objects that are inactive are impacting performance or should I consider removing inactive objects (basically those that stopped moving) from physics simulation and adding them back if needed. For example - what i was thinking is to have some sort of either thread or some other monitoring system that adds simulation to physics object only if they are in some area from the actor (so, basically, only those that can possibly be moved , hit, etc, so that the objects that are a mile away do not simulate for no reason). And for those that are outside of that area, as soon as they stop moving they would be removed from physics simulation and added back only if the actor approaches within some specified distance. (so that they can respond again , if actor were to push them, hit them, etc)

If most of the physics objects are inactive and waiting for energy to be applied are they impacting performance enough to warrant something like this , or is this an overkill and unnecessary extra work?