I approached this by adjusting the input based on the angle like I was trying in my first post:
If the camera angle is between -135 degrees AND 135 degrees OR between -45 degrees AND -135 degrees the input needs to apply to the opposite direction, with an accompanying function for the y input.
This also required a swap to determine which direction the input should apply to:
This works good enough, but I would still rather it work at any direction without having to define the ranges.