Is there a way to set a complex collision to only a certain part of my mesh?

Yes you can do that by placing collisions on your mesh manually. Or you could reimport the model by having it’s wood and leaf parts separate and setting their collisions accordingly.