Based on a few other replies in various posts, I now do this, in my constructor, at the end:
#if WITH_EDITORONLY_DATA
UActorComponent* StupidUnwantedArrow = Cast<UActorComponent>(WidgetInteractionComponent->GetDefaultSubobjectByName("ArrowComponent0"));
if (IsValid(StupidUnwantedArrow ))
{
RemoveOwnedComponent(StupidUnwantedArrow );
StupidUnwantedArrow ->DestroyComponent();
}
#endif
This basically goes and finds the stupid arrow, and kills it.
Perhaps it is not ideal, but it works for me.
The arrow is only added by the WidgetInteractionComponent in WITH_EDITORONLY_DATA code, which is why I only try to destroy it in the same situation.