What is the best way to integrate OpenSteer in my project

Best way is to make plug in, since this way you can easily re use it in any project

since 4.8 there a plugin wizard (Which is plugin by it self, which you need to enable first). UE4 use it own build system and including libraries requires some modifications in build scripts of module or else you will get linking errors:

After that integrate libery in to UE4 framework as much as you can, try to find where this libery belongs in framework and try to attach it to it. That library is MIT licenced so if you do good plug-in, you can place it in UE4 marketplace in future ;] (plugin marketplace is yet to appear)

Also if you gonna make any extra editor features (this excludes standard UFUNCTION and UPROPERTY bindings to blueprint as well as classes that you can make blueprints from it, they all gameplay code as long as you not using editor functions and modules), you need to place them in sperate module, editor APIs are inaccessible in game build and editor code won’t build on packaging of the game, so one module that will come with game and one module that will be only in editor. Or if that possible you can just use #if WITH_EDITOR #endif in game module.

Plug in is like separate project it can have more then one module as well as content assets