How do I use online subsystem for my own API?

i didnt quite understand what you mean though i assume you are saying that you want to host your server on your pc. and you want your own matchmaking logic.

However i will tell you what i know.

take a look at this. this tells you how to create a executable for both the game and the server.

also even without subsystem nodes for multiplayer are already provided in ue4