First, for the walls I found a fix by using the UObject::Rename method to set a new outer for the walls. I assume that the player movement due to the base movement ignore components with the same outer as the base (I tried with the owner but it didn’t worked). Maybe it’s a bit dirty but it works, and since the walls I spawn won’t be updated later, I feel like it’s no big deal.
Last thing I did is to create my own CharacterMovementComponent, override the SetBase method with an empty method, and compute the new player location manually. It seems to fix both the collision between players and with walls (I disabled the solution I mentionned above).
I placed break points in HandleImpact, but sometimes it was triggering and sometimes not, at least never when the collision between players happened 