With that fix(which is horrible, it querys the arrays many times every frame, its temporary but still, eww) and some messing with the threshold, it behaves much nicer!
Threshold helps because no note is perfectly limited to a frequency, they always have overtones and stuff. Look at the spectrum of this sound https://youtu.be/m7TgZbepZtA