Correct; that’s how foley sounds are made. A good and relevant example is this: Making sound effects for video games #3 tutorial - Eldest Souls - Indie DB. He recorded the sounds himself, but nothing is stopping you from getting sounds elsewhere, especially when you are going to mutate them as much as he did; tracing the sound of rain back to frying bacon is gonna be hard.
There was also a lawsuit involving a sample used without permission in Jay Z’s “Run This Town” that was dismissed because:
“‘Run This Town’ bears very little and perhaps no similarity at all to ‘Hook & Sling Part 1.’ The melody and lyrics are entirely different. The lyrics do not contain the word ‘oh,'” Judge Lewis Kaplan wrote in his statement, adding that the “Oh!” was only used in the background “in such a way as to be audible and aurally intelligible only to the most attentive and capable listener.” Kaplan added, “The word ‘oh’ is a single and commonplace word. Standing alone, it likely is not deserving of copyright protection.”
If you know you can’t take the 3D models, what makes you think you can take the sounds?