Virtual Shadow maps with custom vertexFactory/sceneProxy

I’m updating my plugin for UE5. I have almost done it, the only thing that is not working are shadows using the virtual shadow maps. My plugin contains a custom vertex factory and component to do skeletal mesh instancing on gpu. From what I saw from the UE source code I need to fill an array of FPRimitiveInstance and implementing the TArray* MySceneProxy::GetPrimitiveInstances(). On the vertex factory side, I saw the use of the GPU scene and the primitive uniform buffer to get the right data. Because from what I saw only the primitives that survive the new gpu culling pass will be used for the virtual shadow map pass. Now there is a problem in my case. I have already buffers loaded on the GPU with the right instance data since I have a custom compute shader that does culling and LODs selection of the instances in one pass, so keeping and updating the instance array of the proxy on CPU is a waste of memory and time. Moreover, the GPU culling pass doesn’t seem to select the LODs but just culling the primitives. Is there any other way to support VSM without filling the instance array or primitive buffer from the CPU?