Update! I’ve managed to compile 4.18 preview 2 (which has the commit merged), and although menus and title bar buttons are now working correctly in high DPI mode, the scaling is still way too aggressive. Here are the relevant log lines:
[2017.09.27-20.25.20:651][ 0]LogLinux: Scale at X=0.000000, Y=0.000000: 1.750000 (monitor=#0, HDPI=157.825241 (horz scale: 1.644013), VDPI=158.566467 (vert scale: 1.651734))
[2017.09.27-20.25.20:651][ 0]LogLinux: Scale at X=1676.250000, Y=917.625000: 1.750000 (monitor=#0, HDPI=157.825241 (horz scale: 1.644013), VDPI=158.566467 (vert scale: 1.651734))
[2017.09.27-20.25.20:682][ 0]LogLinux: Scale at X=0.000000, Y=0.000000: 1.750000 (monitor=#0, HDPI=157.825241 (horz scale: 1.644013), VDPI=158.566467 (vert scale: 1.651734))
The detected DPI is correct, it’s just the resulting scale that is unexpectedly high. Seems like the DPI scaling math makes sense for desktop monitors which have way lower DPI overall, but it doesn’t really work too well for laptops. Personally, I’d rather have DPI scaling be an user setting on Linux (the only desktop platform without an unified system DPI scale setting, sadly), instead of an auto-detected value, which would cover all possible scenarios.