In the same “zips” folder where the app lives, there’s a zip called “msImprovedWebservers.zip”. Not sure if it’s the same for you with a custom one. By default, that file gets downloaded with each new VM just like the app, so you should be able to replace anything here. Luckily it’s not all baked into the VM itself.
The VMs with the UE app also host the frontend web page, so instructions for refreshing them are the same as above. The matchmaker VM is not scaling up and down fresh machines like the UE scale set, so if you have custom matchmaker code to replace you might need to RDP in the matchmakers, replace stuff, and restart.