VR asymetric spawning in multiplayer

Hi, i’ve recently taken on the task of making a game that will be played with a pair of players with one in VR and one not in VR. I am using steam advanced sessions for the multiplayer and it works pretty well. The only problem is that the game only spawns whatever the server is using. If the server is the non-VR player, then the VR player will get a third person character. If the VR player is the server, then the non-VR player get a VRPawn and crashes. I have tried for more than a week every possible way of going at this, but no luck. I even made it so that the server always got third person and the client a VRpawn just to see if that would work, but no luck, everyone gets whatever the server gets. I am starting to lose hope, hopefully someone know a way to fix this.

(The project is in blueprints btw)

where and how are you spawning your pawns?

I have tried many different ways, but my current system works like this:
-Player controller checks if VR using the HMD enable node and sets a variable with the class of the corresponding pawn

-Gamemode adds all player controllers to an array then runs through that array, casting to the player controller and getting the pawn class variable, spawning it, and possessing the pawn with the correct player controller

The only problem is that the client gets whatever pawn the server gets