Ok so this is a really important detail. I did a lot of optimizing of the RPCs and found a ton of inefficiencies. But I found that the problems didn’t all go away.
I struggled for a bit with this issue for a bit and even started losing hope that I can get this to work as a multiplayer project.
Well as a last ditch effort I made sure to run a test where a client connected from another machine rather than my own. Instantly the issues all vanished. So apparently, testing performance on one machine is going to give bad results. As soon as I had another machine connect to the server I was suddenly able to see everything working as its supposed to work.
The lag issues are a false alarm now.
As a result I will put up a test build to show progress tomorrow!