Muouse Input Disabled When Switching to Game Only Input

When switching to Game Only Input Mode there is no longer any mouse input. I had tried the other input modes too, but the mouse is still not working.
I use left click to begin attacking for example, so when I am switching from menus back to the game the mouse stops working.

Is there any way to prevent this from happening?