From another thread I found how to display motion vectors: Gen 5 Temporal Anti-Aliasing - #15 by Guillaume.Abadie
In the editor window, click Show → Visualize → Motion Blur. That showed me that the only mesh generating motion vectors is the regular static mesh (the middle in the image). The instanced static meshes do not.
I found that the smearing improves a lot in this test project by using TAA instead of TSR. Just the switch to TAA did not resolve the issue in our own project from my original post, so I tried applying the same logic I found in this test project.
We had “Output velocities due to vertex deformation” enabled, which generated motion vectors for our instanced static meshes, which did not happen in the test project, so I disabled this. That improved the smearing to the point where transform changes did not cause any smearing, but since we turned off the motion vectors from WPO changes, whenever our enemies animate, the smearing is quite bad.
In essence, fixing one problem causes the other and vice versa…