For others that come to this post looking for an answer, the fix is moving the project off iCloud drive. i.e don’t have them in a folder that is getting stored on iCloud (like the Documents folder). It seems the “detritus” is metadata iCloud keeps adding to the files as it syncs them. Given this is part of Apple’s ecosystem I’m surprised this is broken in XCode 
.