Those widgets need to be a part of the widget that is being moved / zoomed.
Zooming or panning is generally done by nesting a canvas within another canvas and shifting the inner one around - not that dissimialar to dragging.
Remember me nagging about adding widgets to a canvas? I think I even mentioned moving the whole thing at that point - that’s how I would approach it.
The waypoints would need to become children of the container that being transformed.
I have a thread about zooming and moving a map out there, with a project attached. May be worth a look at in case you get stuck.