Overlap collision object still behaving like a solid

I have created a collision mesh inside of an object and set its collision type to “Overlap All” however when my ship collides with this object it still behaves as if it is solid (My pawn can not fly through it and bounces off in the other direction). I can’t seem to find anything about this and from watching tutorials it seems to me like this behavior should not be occurring. Does anyone have any suggestions. The other object in question is the default flying pawn from the flying tutorial. Thanks.