Ok, figured this out.
For some reason, none of the presets work for me in colliison.
However, setting the Collision Preset to ‘Custom’, Collision Enabled to ‘No Physics Collision’ and then object type to ‘Pawn’ made it work.
If I set to ‘Overlap All’ in the presets, which has the exact same flags, the event does not get triggered.
this will work for now, however, let me know if this sounds like a bug to anyone. At least at this moment I am no longer blocked.