For anybody in the future having this issue.
Check the Near Clip Plane setting and increase it. Mine was set to 1 and it seems that this was causing the issues. After increasing it the artifacts were gone.
Would love to help anybody in the future who stumbles to this problem 
2 Likes