Found a solution finally! My error was caused by Garbage Collector who destroyed/invalidated my UObjects silently in the background, thus pointers became invalid or set to null/uninitialized after a while.
My TArray<> and TMap<> structures inside InventoryComponent were not decorated with UPROPERTY() so GC knows about them.
Solution was to decorate them properly:
I did not know we need to decorate containers so GC knows about them 
Hope it helps someone else!