yeah bit tricky with the transforms. world to local, add the offset, local to world, and subtract world position.