I wonder if you’re better off not using gravity at all but just applying a force in your chosen gravity direction at all times and switching that direction as your character changes their position. I’m assuming you’re doing a Prey style game where you can walk sideways on a wall or upside down on a ceiling sometimes.
I know I am not quite answering your question but I would try it this way if you don’t find a better solution.