Use Object Types, like in the image above;
Everything that should be ignored is set to a different Object Channel group.
I use them in my game to detect which objects can be manipulated or not; Whatever is from the WorldStatic group for example, the Line Trace will ignore.