Client with default movement component isn't moving correctly on server

I have a really weird bug where the clients location seems to be “unsynced” with the server. This only occurs when I open a level with a random generated dungeon. The way it works is the server beforehand calculates where the segments of the dungeon should be and then when the level opens the server spawns and replicates the segment. This generation of the dungeon does work. When I the servertravel to the level, clients have issues walking around; on the server it looks like they are walking into walls and keep getting stuck, then the clients get teleported to the location where they are from the servers perspective (which isn’t correct somehow).
I thought the issue may be in how the dungeon is spawned but I can not find a solution. I tried spawning only on server and setting the segments to replicate - movement still doesn’t work correctly. I tried spawning on server and on clients locally - doesn’t work either. I can see the dungeon on server and client so it means that the replication did happen but clients can not move around correctly.
I also prebuilt one dungeon map and this does work fine (same segments as the random generated one) but this wouldn’t be random generated any more which was the fun part about the dungeon.

Does anyone have an idea why this problem occurs?

Notes:
Default movement component
No landscape in the random generated dungeon level
No errors in console
Only occurs when the segments are spawned at runtime
I tried making a function where it sets the client location on server from the clients perspective but this is even more laggy and doesn’t work
Programmed in blueprints
Dungeon segments saved beforehand in game instance (on both client and server)