I think it would be helpful to look at the build in navigation system. Is it possible to optimize away all the empty voxels and rebuild them if you need them?
I know it is a huge undertaking but making this plugin behave more like the build in navigation system would be so awesome.