Disabling event driven loading did not change anything unfortunately. The same issues and crashes still occur.
There is defenitely something wrong with phys engine / phys materials, sometimes the fatal crash is a bit slow, and there is time to observe how all phys materials in the entire level returns errors/access none, but not untill after a few server travels.
My method to reproduce the issue when I test it is:
Load level 1, run around a while, servertravel to level 2, level 3, level 4, and back to level 1, then run level 1 untill physmaterials break, and eventually fatal error crashes.
Edit: Im still not sure what triggers the problem, could be many factors.