Listen server: when host loses focus the clients lose net update

I’m working on a coop game that uses a lister-server host. The problem I’m facing now, and I’m not sure how to fix (my google-fu has failed me), is that when host loses focus of their game (alt-tab, for example), the clients net refresh rate get tanked. Server game probably gets throttled and runs at ~5 fps and that’s the rate of updates the clients get, which causes all sorts of problems.

So, any tips how I could tackle this issue?