Before publishing my game, is there a way to upload a custom thumbnail picture?
Hey there @YTSTARLIGHT! Welcome to the community! You can change your thumbnail in the Project Settings->Description:
However, if you mean your windows icon, you can find that here: Project Settings->Windows