Okay this might be the weirdest answer ever, but I randomly switched the cooking settings to shipping build which brought me around +30 FPS and now it seems to be a GPU Time bottleneck since DLSS gets me around +50 FPS. This is the weirdest behaviour for me but it works I guess… makes optimization harder tho xD