Is there a config option to enable higher resolution for retina displays?

Can you try launching a project directly? See guide here: Running Unreal Engine | Unreal Engine 5.1 Documentation

You’ll need to click on MacOS at top and then go to “Launch Editor from Command Line”