Bricks physics simulation, is this feasible?

Hi everybody,
I was doing some experiments today with ‘real’ construction pieces like bricks and I just wanted to create a realistic wall made of them.
Physics simulation is enabled and so is gravity and collision for the bricks. They have a standard physics material which is concrete (and actually this doesn’t change a thing).
The problem is, as soon as I hit simulate, the construction becomes wobbly and crumbles in a quite ridiculous way. Sometimes the pieces on the floor start bouncing rhythmically too.
You can see this all happening in the gif below.

I don’t understand this type of behavior and what I am asking you kind people is if:

-is a realistic physics simulation of something built with single pieces like this even possible?
-do you know any workarounds or should I just forget about this?

Thank you so much in advance, I hope this is interesting for others out there as well. Bye!