Is there a square brush for landscape painting?

I’m pretty sure this isn’t an option. You can usually get the desired effect by adjusting the density, radius and other settings on each foliage type you’re using. If that’s not enough control, you can remove instances that aren’t placed to your liking using the single instance selector (then you can move them around or delete them).