It uses the ear clipping algorithm. The source code has this comment:
// NOTE: This polygon triangulation code is partially based on the ear cutting algorithm described on
// page 497 of the book Real-time Collision Detection, published in 2005.
The book it talks about is this: https://amzn.to/362dCHj
(Which I think should be required reading for any 3D game programmer. It’s great!)
Note that there’s two implementations of essentially the same algorithm, one in FMeshDescription::ComputePolygonTriangulation() and one in FGeomTools::TriangulatePoly().