Update: On 4.25 the main issue is fixed and now the Automation Driver successfully takes control and simulates the input. There is still an issue, where the tests fail the first time. The error is the following:
Failed to load cursor ‘…/…/…/Hannibal/Content/Slate/Cursor/DefaultCursor’ [C:\gamedev\hannibal\main\ue4\Engine\Source\Runtime\Engine\Private\GameViewportClient.cpp(3776)]
It seems the first time it’s trying to load a Hardware cursor through the AutomatedApplication::CreateCursorFromRGBABuffer which is not implemented and returns a null which then gives the above error.
It appears to be an issue of loading the test level/game after the test has started whereas to my understanding it should be the other way around.
So to bypass this issue as a temp solution, run the test once, keep the PIE window open (should stay open on its own), run the test again and it works.