Found it, it doesn’t work.
It was nonsense from me though that disabling TSR didn’t help, it solves the issue actually, but it’s quite disappointing that the setting doesn’t work (it says it’s there to solve exactly this issue in it’s tooltip).
The only solution seems to be to use MSAA or FXAA. TAA is also affected by this issue.
Also, it doesn’t just happen when velocity writes during base pass, it happens when it’s written on depth pass as well.
I recorded a video but couldn’t find a suitable site to upload it, it’s a shame ue forums doesn’t support videos.