You kind of demanding too much from engine. Wait about 10 years with this idea, else you are for some serious pain.
Or optimize/cut everything possible.
Forget dynamic shadows from thousands of physic actors. I think you also should forget dynamic lighting for them, instead make unlit material with fake global light vector, and some static cubemap reflection for shiny spots.
Or you need some very clever C++ code that updates static mesh of wall/tower makes bricky hole where it should make and spawn few bricks. Then you can do some better lighting. You could do this by keeping 3d voxel shape of castle in array, then update mesh from it. But for that forget blueprints.
So either you do hard work or players gpu does.