Getting separation of controllers in the X and Y axis

Im setting up a system where a player can point at a specific type of actor, hold the grip buttons on both controllers and then move the controllers to scale the object (apart increases the scale and vice versa). This works fine until the player turns to their right and can no longer make the object wider.

The way it works currently is that i get the relative locations of the controllers when I press the grips and subtract them to get a separation. Then, on tick, I get the current separation of the controllers, subtract the initial separation from that, get the two axes I want from that vector (the actor only gets scaled in two axes) and add that on to the original scale of the object. I have also tried projecting the new separation vector onto the up and right vectors of the controller and then grabbing the elements I want from the new vectors and make a vector out of those. This method almost works but the increasing width is the opposite way the controllers move when the player turns to their right.

Is there a better way to do this kind of thing or get the current method to work?

Thanks in advance

Well, i did something like that with Unity and LeapMotion. When the hands separates the scale of the object increase, etc. I compare the size or magnitude of the distance vector between the two hands, each frame.
If u dont want to depends on position, maybe u can work with size of vectors and not a initial locations of controllers but with locations of controllers each tick. Good luck!