Man how you can run multiple instances?
and for your problem you need to know that the video cards has a lock from factory in then [NVENC] controller must cards has 3 to 4 instances on the driver
but you can patch your drivers to unlock the number of instances that can run at same time look at this GitHub - keylase/nvidia-patch: This patch removes restriction on maximum number of simultaneous NVENC video encoding sessions imposed by Nvidia to consumer-grade GPUs.
this is the cheaper answer the pro answer is buy a nvidia quadro card you can look more about the models and the max concurrent sessions on this web
Video Encode and Decode GPU Support Matrix | NVIDIA Developer