Really bad performance when Nanite's on even when there's nearly nothing in the viewport

So I have a really massive openworld environment (let’s say a real city) with many SM actors. When Nanite is enabled I noticed that when my camera is zooming into really close to the ground, the FPS is still very unsatisfying (only 20FPS with i9-13900KF CPU and an RTX-4090 GPU). When I disable Nanite using r.Nanite 0 in the console, the FPS goes up. I noticed in my case when Nanite is on, the RHI thread is the bottleneck. So I use the Session Frontend tool to capture some stats for profiling. The results are as follows.


But I just did not quite understand it. Can anybody help me please.

[Update] I managed to alleviate this issue by setting the CVar r.Nanite.AllowProgrammableRaster=false. This will help but the SMs with musked material or WPO and PDO feature will not function properly. I guess I will have to re-design the materials used in my project for a better material practice with Nanite.