What you have to reference is the soft class, so you can spawn the bp with the logic you want. When you want something from the inventory to be used in the game you look at the class and spawn it
You are mixing the logic of the object with the logic of the inventory, an inventory should have only the ids of the tables.
Only when you want to use it do you spawn them in the game
If a blueprint uses widgets, what I do, let it be that Bp that manages the widget including the spawn, and add to viewport, so what you say never happens, there is nothing more terrifying than a widget manager hahaha