Destructible Gets Fractured by Character Collision

My destructible mesh fractures and takes damage if my character collides with it. How to avoid getting damage from player collision and only fracture when hit by bullet?
I have tried to increase damage threshold and I have tried to set the mesh to ignore the player but that made the player walks through it, I want it to block and damaged only received when collided with the bullet.
Bullet works fine but the player can also fracture it which is funny and weird!

You have to attach some type of collision(box/capsule/sphere) to the destructible mesh. Then set the collision to Query Only(No Physics Collision), and Ignore everything except Pawn, set it to Block. Sorry for such a late response.