I understand the problem at hand. What I’m saying is problems with the mesh will create improper shading. Badly smoothed geometry is a common culprit. This is because game engines will interpolate the vertex normals from one point to another. So if you have two vertices with very different normal directions that are physically spaced apart, then the interpolation between these points can create unattractive shading.
Good mesh geometry is critical to getting proper shading. One common cause of bad shading is when edge smoothing is enabled on large surfaces that have steep angles. They should either be marked as sharp, or supporting geometry can be added to interpolate the normals and therefor the shading better. This type of artifact is clearly visible in the left edge of the hole.
90% of weird shadows are due to bad mesh normals, and/or bad normal maps.
Its possible it is something else. But since it happens when shadows are disabled it almost certainly isn’t a shadow. To explain further we need to clarify how shading is different from a shadow.
Shading exists on any lit object as a consequence of how much its face (normal) is pointed towards the light - whether it has a cast shadow or not. The dark part of this self shading is also known as an attached shadow in in life and art. Cast shadows are when an objects shadow falls on another surface (such as the roof surface casting the diagonal shadow on the wall you mentioned earlier).
Disabling shadow casting will never remove the attached shadows/shading. Changing a materials shading model to unlit will remove its shading - and most likely your artifact. Unfortunately it will remove all the shading that you do want too, so it isn’t a solution.
Its also possible that the shading is correct for the angle of the sun. Try rotating the main directional light and see how the surface reacts. If this is the case, you would need to add more light, such as increasing the skylight intensity.