I finally found an issue, it was because when importing the Revit file I checked “lighting”, I thought it means light fixtures/sources but it seems that it is actually importing the lightmaps and as I have thousand of them it caused a serious performance issue. I unchecked the box and everything works now.
But now I am having an issue that I all imported meshes are no colliding no matter how I tweak it so I am still trying to figure out…