I’ve never noticed such a drop and I use quite heavy UI also so I guess you must be doing something wrong.
Could you paste your widget initialization code and maybe check if same things happen if you just add 1 button or any other custom widget to your UI?