I need to do some more testing but I think I actually just got mine working. Rather than using UI only as the input mode, I swapped it to the Game and UI and then unchecked the mouse and cursor options, which seems to keep it from rotating the camera when not hovering over the inventory. I will post later today after I run more tests to verify this.