Widget instances are being created even when not loaded

Are you perhaps creating a multi-player game? If so, your widget may be replicating and loading with your other players. Check here:

Why is this widget not hiding? - Programming & Scripting / Blueprint - Epic Developer Community Forums