So, I did some testing with an extra account in a published version and it seems that the trackers are working as intended once that option is checked.
So, that leads me to question when that option was added or if the default setting changed at some point because I never had this issue before. I’ll try to test a bit more but for now I will hope it’s resolved.