I have placed a tiny amount of actors into the level but the size map shows all kinds of references:
99% Of the references have nothing to do with this specific map as they weren’t placed there.
—> Are the blue bars on top of the size map relevant? Or should i only look at the right side (pink, green, turqoise)?
Everything in there is relevant 
If you don’t understand where the connection is, take a look in the reference viewer:
The reference viewer doesn’t really help. It shows there is an alleged connection between the two assets, but doesn’t actually tell you what that connection is. I removed all references to a particular class from a UMG widget and both the size map and reference viewer are claiming a connection still exists. I can’t find one anywhere. I cleaned up for 4 connected classes to that widget, 3 of them disappeared, one didn’t.