Weapon-only collision making player jitter, player-only collision not allowing pickaxe to pass trough

I want the purple cubes to not collide with the player but detect pickaxe hits, they are set to the collision preset “FortTriggerOnlyWeapons”. I want the white cubes to only block the player and let the pickaxe pass trough them, they are set to the collision preset “InvisibleWall”.
However, while the player can get inside of the purple cubes, movement inside of them is really weird and jittery, and you can’t jump. Also, the pickaxe doesn’t pass trough the white cubes, it either completely nulls the pickaxe hit (in cases where I’d want to hit something behind the white cubes) or gets cancelled instantly (same issue persists with preset “InvisibleWallDynamic”).