Camera Rotation requires "Set Input Mode Game Only" every time, otherwise..

Hello, I am building a RTS style camera rotation for my camera-player pawn, where the player can rotate the camera by holding a key and move the mouse - and it disables the cursor while doing so.

Now to be able to rotate the camera, I have to first “Set Input Mode Game Only” every time I rotate the camera. Otherwise it requires me a second mouse-click (while holding the rotation key) for it to work the first time.
After that, it works normally by just using the rotation key.

Obviously, if I don’t set the input mode every time, it’s very confusing why the camera rotation is not working as intended (and it looks like bad programming if I say “you have to press a mouse key while holding the rotation key for the first time for it to work”).

Can someone explain to me how this is not a bug, or what is a fix for this? It seems weird to me that I have to set the input mode every time for it to work right away.