I am also having this issue, since 5.3. Sometimes EndFlushResources will take less than 0.5 ms, other times it will take 5ms and remain that way for the duration of the game. The only thing I found during testing to reliably trigger it is executing more than 1 task on the task graph from the game thread.
Here’s a picture of my profiling and annotations.
I’ve seen some other plausible explanations such as losing and regaining window focus and vsync/other sync configurations-- though I have not found any success looking into these.