VR Expansion Plugin

Should be something wrong on your side, its kind of hard to know without visualizing what you mean by “swing around”. Could be them attached to their parent with an offset, a physics collision jitter, a late update retention, or something else.