Figured it out. I now also implement PostLoad and build the components there. I then track if any of the important properties have changed and only do another build if they have, so it will only build the components once regardless of when I try and build them. It works both in the editor and when playing, and I get the effect that as the BP properties are changed (such as Width) it automatically destroys the old components and builds new ones.