Thanks for the suggestion re sleep/awake workflow! Makes a lot of sense to me.
Will look into it and keep you posted!
Alright. I can pretty much confirm now that the glitch is caused by slow moving physics objects. If I disable physics the moment the object drops under a certain value for linear and angular velocity, I can get rid of the error almost completely. Therefore I assume that this is what is causing the issue.
Unfortunately I do kind of want to avoid loosing physics completely as the whole thing depends on it, so my best workaround at the moment is to turn physics off and on again immediately, virtually stopping the object but still have it be responsive in case it gets hit by other actors. Not ideal, as it still introduces a good amount of glitchiness, but the best I could find for the moment.