How to have multiple characters carry a single object?

I’m working on a game similar to Pikmin, where you have little minions carry around objects for you. Currently, they carry objects using a physics constraint tool attached to them. This works when its a single minion carrying an object, but when multiple do this at once it messes with the physics and makes the object being carried freak out. what alternatives are there to physics constraints, that would work for having multiple characters carry a single object? alternatively, how can I alter the physics constraint properties so its physics don’t break like they currently do?