If you remove the widget from the viewport and do not store the reference, you will have it (and everything in it) destroyed by garbage collection.
Consider the following:
When you hit Tab for the first time, you have no widget - it’s Not Valid, so a widget will be created, reference stored, it’s made Visible and gets added to the Viewport.
Next time we hit Tab, the widget isValid, we switch between visibility states.
However, in most cases, it makes more sense to do this:
Create the inventory when the game starts and then simply show / hide it. What if the player runs into a pickable item before even opening the inventory? They will have no place to store it. I’m simplifying here but it’s worth considering.