@NasteX, Thanks so much for your answer!
I’ve already played around quite a bit with texture streaming and that doesn’t seem to be it. In fact I gave it a go throwing out all textures in the scene and it still does it. It does get a lot less, but I assume the normal map simply makes it more obvious as it adds more detail.
My best guess right now is that the TemporalAA glitches out when an actor moves incredibly slow? Is that possible?
All of those coins are dynamic movable actors, so there is quite a bit of overall movement going on in the scene. My impression is that the noise seems to come in on the coins that are veeeery slow to almost not moving at all anymore.