I love the idea.
You’ll probably want to think about HRTF sound as part of an LOD type of audio implementation. In this way, you’ll want your HRTF sources to be mid-to-close distance from the player; but at great distances, it won’t be appreciable. In this way you might want to abstract sound sources with zoning so that a group of trees at a great distance sounds like a distant forest, etc.
You will also (believe it or not) want to take advantage of non-spatialized tricks like flat binaural recordings or well resolved stereo sound events.
You’ll find that the filters on some HRTF renderers are quite harsh, and result in a very precise point source which might be undesirable for large sound sources. In those cases, you might find it more effective breaking the sound up into elements, some which are rendered via normal panning and some which are rendered via HRTF.
In 4.18, you’ll find HRTF renderers for both Steam Audio and Oculus Rift plugins, so you’ll just enable one of those for your project and select which one you want to use as spatialization in your platform settings.