You could start with a description of your kit. What CPU? What GPU? How much memory?
What platform? Is the project desktop,Vr,Mobile?
By drop in frame rate by how much?
Are you using complex shaders (Press alt 8 and if you see a lot of red that’s bad. White extremely bad)
It should be noted that FPS is no longer a reliable means to determine performance loss as say compared to refresh rates and GPU profiling.
A drop in FPS “could” indicate a CPU bottle neck where GPU profiling will tell you if you GPU is the bottle neck
Best practice wise as to figuring it out start here
Do a proper profile as demonstrated you will be able to pinpoint the problem and not just say, think, your problem is related to performance drops when you look at something then improves when you look away. Doing so while working in the PIE is common and such testing for problems should always be preformed configured as to how the final result will be used.