Me again! So the installed version crashed even when I went back to normal Shipping mode with Distribution unticked, which was weird as that was working fine before.
Turns out that removing the Google Play License Key actually did the trick, so installing direct to your phone with that included seems to cause a crash.