For Sweep to work, the colliding element must be the root, not attached to the root.
Pop it in and it should work. 
Fair enough - most control this way. However, in case you wanted actual physics involved, note this component:
It is quite happy to overlap its field with other components and quite capable of generating Lagrange points, for example. When Impulse Strength is set to a negative value, it acts as a gravity well. Ideally, you’d run this during Async Physics Tick.
I am then moving the player with Add Actor World Offset, and don’t understand why it’s not able to collide.
Depends on what we’re really trying to achieve - perhaps not the right tool is being used. For a controllable pawn one would take advantage of a floating movement component which would then produce superb sliding collision results.