So there’s another undocumented issue people are probably encountering…
You can set your .ico file in Project Settings but the Editor will copy and reference it into your Windows build folder as Application.ico.
For people that package directly in the ../Build/Windows/ folder, this creates an issue. The copied icon would get deleted on packaging, thus have nothing to reference.
So make sure you create a subfolder to package into like so: ../Build/Windows/shipping/.
You should see the icon copied and renamed in: ../Build/Windows/Application.ico.
Your packaged game exe’s will have your custom icon with no additional floating icon files.