I came up with a workaround for this issue, I also have a vehicle and a character that can drive it.
What I did was have a invisible copied mesh thats exactly the same as the vehicle mesh and have that to be kinematic, then in the tick update its location and rotation with the real mesh and have the character only collide with that copied mesh instead of the physics mesh.