World space widget not working in multiplayer

I have a widget in the world space that is component of a blueprint actor, I have set a widget interaction component on the character BP and it works as intended with one player. I have one blueprint actor with the widget, set up in each separate room that the players spawns in. I have the widget components not set to replicate as I want whatever the player does on them to be local and once they press a button then the logic for that would be sent to the server. Once a second player joins neither player is able press any button on their widgets. Additionally the buttons no longer even change appearance on hover once the second player joins. I put on debug for the widget interaction and I can see the debug line is there and stopping at the right spot. I am not sure if this is due to the widget interaction breaking in the multiplayer, or if it is due with some replication issue with the widgets?

When setting up a world space widget are there certain settings I should change once multiplayer is involved? (Outside the actual logic from the buttons being replicated)