Spent the last hour or so messing around trying to understand what was wrong, turns out I missed changing the collision settings for the mesh for my character specifically.
For future reference; make sure all parts of your character have the collision settings set to your custom channel!