Implement closing inside the inventory:
onKeyDown
Input Mode Game & UI
Also, EI supports context mapping with priority. Nothing stops you from adding an entire new input scheme that works only inside the UI and consumes all else. As in: