As of OS X 10.9, OS X now supports OpenGL 4.1 and along with it GPU accelerated tessellation.
Looking through the codebase, under OS X, UE4 explicitly requests an OpenGL 3.2 context. What would be nice is under systems that support OpenGL 4.1 it’d request a 4.1 context instead.
We want to support OpenGL 3.2 cards as well, so we use features added in Mavericks, including tessellation, via extensions (see FMacOpenGL::ProcessExtensions())
The context flag we use requests an OpenGL 3 or higher context, so we actually get a 4.1 context under 10.9 and as Michael points out, we do expose the extensions through the FMacOpenGL class. Unfortunately in order to use in the current UE4 OpenGL implementation we need some 4.2 & 4.3 features/extensions that aren’t present on the Mac (yet). This is something I intend to look at & get working, but there’s no timescale.