Same problem.
with 5.0.3 package and 5.1 editor window everything works normally but with 5.1 package the button no longer work.
(I think there is another problem. UE 5.1 packaging only start map)
https://forums.unrealengine.com/t/ue-5-1-package-project-packaging-only-the-game-default-map/692704