For anyone still wondering - this is an expected behaviour.
Construction script will change any default properties for any actor instance and saving the World (i.e. your level) would save those changes in the World. As someone mentioned it is executed on instance creation, means when open your BP, or place your Actor in the world, or when it is loaded with the world etc. If you just clear construction script it would do nothing, because already applied and saved changes not going anywhere. If you need to revert what you did, make explicit opposite changes in construction script.
Although I would not recommend doing it that way. The more proper way would be to create Bluetility with all necessary operations and run it over the necessary objects of the world.