im not sure if this is the issue, but has authority does not mean it’s the server. it could mean that the actor was spawned in the client. people tend to use it because “it smells similar” but then get into issues like this.
isServer or netrole is a better check.
also attaching an actor is a weird way of doing this, but also somewhat clever way to solve the replication.
can you share more of your code? it might well be that the way you’re doing it has a logical bug.
if you do a search you’ll find many posts in this forum about how to best replicate vr components. you should give it a read.
https://forums.unrealengine.com/search?q=replicate%20vr