Can you open the DM and grab a screenshot of your DM settings and post here? Use the eye in the top right of the details panel and select “Show Only Modified Properties.”
Likely there are some values set here that require a lot of damage and multiple hits to destroy the object. Without knowing these settings it’s impossible to narrow down exactly what may be going on since you mention that it does destroy on impact with the character and after a lot of hits. Possibly damage cap or damage threshold is higher than you would like.
Let me know.