I have the same problem as you, I think this is a bug, I am also player 1 using mouse and keyboard, player 2 using game controller.
I found that this problem occurs whenever the first player is keyboard and the second player is gamepad, and Skip Assigning Gamepad to Player 1 is set. This problem does not occur when another gamepad is inserted (two gamepads) and the second gamepad is used to control the second player, but this forces the player to select two gamepads.