I was getting similar slowdowns when my character blueprint got over crowded with nodes, ended up splitting functionality into separate Blueprints to reduce the problem.
At its worst state I would wait for 1-2 seconds when trying to connect, delete or modify any connections or nodes in the graph.
I have a GTX 650, and a not a very powerful rig, but its doing an Ok job for the most part.