Sorry, I think I’m at about the limit of the help that I can give remotely. It doesn’t seem like you’re doing anything obviously wrong, but without my hands on it… well… really sorry.
The only other option I can think of would be to move some parts into C++, but that may not be an option for you.