I had hoped taking the dot product of these two would return a positive or negative value/angle, but that didn’t work, I assume since the vectors are always going to be pointing in roughly the same direction, they’ll never be negative. And I’ve been messing around with the cross product but I don’t see how getting a perpendicular vector would help me. I’m at a loss as to the best way to discern the difference here.
Try the dot product of each normal to the direction vector. Direction vector would be point1 - point2, normalized. If both products are positive, it’s concave as the vectors are “pointing at each other”. If both are neg, it’s convex. Mixed would indicate a nearly flat surface.
Edit: for point2 dot product, negate the direction vector… P2-P1
It’s the hit locations. Hit 1 location is point1, hit location 2 is point2. P1 - P2 gives a vector that points in the direction between points. You want to check whether the normals are pointing at each other using this
You can use the Z of the cross product between V1 and V2:
V1 = t1 -t2
V2 = t3-t2
Z = (t1.x - t2.x)*(t3.y-t2.y) - (t1.y - t2.y)*(t3.x-t2.x)
(Note that this is not the general case. There are shapes in 3D like hyperbolic paraboloid that are both convex and concave depending on your reference plane.)
You can also use the cross product of the two vectors and check its sign against a reference up axis. If world up is (0,0,1) and both vectors lie in the horizontal plane (XY), then Cross(V1, V2).Z tells you whether V2 is to the left or right of V1.
Wait, forgot to normalize. One moment. (Edit: Fixed. Normalizing the direction vectors still doesn’t seem to work so I’m assuming there’s another problem)
But it seems that regardless of the surface, the new dot is always negative and the old dot always positive. Meaning it interprets all walls as flat.
Now I expect, especially when on large walls, for it to sometimes read flat when it’s on a curve, but the Concave and Convex print statements never trigger.
