The easiest way to do it would be to have a separate button for each part of the color wheel.
So to the user, it just looks like a color wheel, but we secretly know it’s a lot of transparent buttons arranged in a circle.
Then you can just set the color by which button they press.