Many of my images were exported from a drone video. Instead, I imported the video through Reality Capture and it now textures and colors properly.
It would be good to have additional logging on what image is an issue should things like this arise. Out of hundreds of images I had no idea where to even start looking.