How do I determine if two hit normals are derived from a concave or convex surface?

I need to determine if the player is wallrunning on a convex or concave surface, ideally by comparing two hit normals like so:

I had hoped taking the dot product of these two would return a positive or negative value/angle, but that didn’t work, I assume since the vectors are always going to be pointing in roughly the same direction, they’ll never be negative. And I’ve been messing around with the cross product but I don’t see how getting a perpendicular vector would help me. I’m at a loss as to the best way to discern the difference here.

Try the dot product of each normal to the direction vector. Direction vector would be point1 - point2, normalized. If both products are positive, it’s concave as the vectors are “pointing at each other”. If both are neg, it’s convex. Mixed would indicate a nearly flat surface.

Edit: for point2 dot product, negate the direction vector… P2-P1

For clarification, is the direction vector player velocity in this instance? Or is this something I need to derive from the hit result?

It’s the hit locations. Hit 1 location is point1, hit location 2 is point2. P1 - P2 gives a vector that points in the direction between points. You want to check whether the normals are pointing at each other using this

It is impossible to be done with the data from a single hit.

The minimum amount of traces is 3 in the plane (t1, t2 and t3). You need to get the angle between V1 and V2:

You can use the Z of the cross product between V1 and V2:

V1 = t1 -t2

V2 = t3-t2

Z = (t1.x - t2.x)*(t3.y-t2.y) - (t1.y - t2.y)*(t3.x-t2.x)

(Note that this is not the general case. There are shapes in 3D like hyperbolic paraboloid that are both convex and concave depending on your reference plane.)

You can also use the cross product of the two vectors and check its sign against a reference up axis. If world up is (0,0,1) and both vectors lie in the horizontal plane (XY), then Cross(V1, V2).Z tells you whether V2 is to the left or right of V1.