Dragging a lever based on orientation & viewing angle

Perhaps I’m not explaining myself well enough…
So here’s a list of scenarios, and how I’d like the mouse to control them:

When the lever’s on the FLOOR and you are in FRONT of it:

  • If you move the mouse UP (-Y), it should rotate away from the camera
  • If you move the mouse DOWN (+Y), it should rotate toward the camera

When the lever’s on the FLOOR and you are BEHIND it:

  • If you move the mouse UP (-Y), it should rotate away from the camera
    (which is the opposite world-space direction of when you are in front of it)
  • If you move the mouse DOWN (+Y), it should rotate toward the camera
    (which is the opposite world-space direction of when you are in front of it)

When the lever’s on the FLOOR and you are BESIDE it:

  • If you move the mouse LEFT (-X), it should rotate to the LEFT of the camera
    (which is the opposite direction of when you are on the other side of it)
  • If you move the mouse RIGHT (+X), it should rotate to the RIGHT of the camera
    (which is the opposite direction of when you are on the other side of it)

When the lever’s on a WALL and you are in FRONT of it:

  • If you move the mouse UP, it should rotate UP (toward the sky)
  • If you move the mouse DOWN, it should rotate DOWN (toward the floor)

When the lever’s on the WALL and you are BESIDE it:

  • Same as when it’s on the wall and you are in front of it

From what I understand, simple vector math should be able to solve this, but I just can’t figure out the details.
If it helps at all, I know the world-space (XYZ) position of:

  • Lever Pivot Point (Base)
  • Lever Handle (Top)

… And of course I know the 2D mouse coordinates, and have the ability to convert any screen-space coordinates to world-space (and vice-versa).

Can anyone help me fill in the blanks, here?