Disable physics and Attach Component to Component for as long as you move the object. Would that work for you? Or is this like a hollow box / bowl inside which you want a simulating mesh to rattle about?
Besides that, you may need to add more details, rephrase or demonstrate visually:
I lower them into the texture of an uneven surface
Not sure how to interpret that. I reckon you forced a physically simulating mesh into another geometry and the depenetration mechanism kicks in and ejects it; that’s to be expected.