Its not supposed to give you an fps boost, PSO caching works by first creating a list of all the needed Shaders that are required by the [Materials] that are used in your UE project. This list is then used to help speed up the compiling process of these Shaders when they are first encountered by your UE project. This, in turn, helps reduce any hitches your project might encounter when a Material requires a new Shader to be compiled.
So your game should now run smoother with no micro hitches, your players will thank you for it.