I really don’t know why, but making a shipping package from the normal menus doesn’t actually do what it says.
You need to use the Launcher from the tools menu, then everything gets baked, and this won’t happen