This was the document that guided me to turn it on and cause all my problems:
https://80.lv/articles/full-photogrammetry-guide-for-3d-artists/
"Camera grouping by EXIF
There is no reason not to use this setting. Enabled grouping can help you avoid the wrong camera and lens estimation. RealityCapture calculates these parameters from all images in one lens group. I strongly recommend this for “one camera with prime lens” turntable scans. Grouping is not necessary for multi-camera rigs."
Other than that issue, that guide is pretty amazing and I would recommend it to anyone getting into or even a vet in photogrammetry.
I am just not sure why it would cause so many problems as it should only really be a good thing for accuracy. I guess the black magic in RC is just far superior 