After looking at the code in FDeferredShadingsceneRenderer::ShouldRenderVelocities, I found out that the Velocity Pass will always be enabled if r.VelocityOutputPass was equal to 0, which means Renders during the depth pass. This splits the depth pass into 2 phases: with and without velocity. .
So I changed r.VelocityOutputPass to 2, made sure Motion Blur, TAA, SSR were disabled, and now it looks like it working, I don’t see any VelocityPass in Insights. This setting is also availble from the editor in the Project Settings under Engine - Rendering.