Look for the desacceleration parameter on the fly section of Character Movement Component. It must be on zero put it on high number or play with to get the point you want. I had the exactly same problem with a flying project a started a few months and that fixed it.