I found the solution to the problem. When calling Set Rotation it is also needed to add the initial rotation of the bone to it. As each bone is its “own rotation space” after getting the world rotation you have to add it the initial bone rotation to transform the world rotatition to the bone rotation