How can I trigger GameSession::RegisterServer in dedicated server in PIE mode?

If you mean custom online subsystem it won’t be that easy of a task. Source look through that forum post for reference I myself referenced it including this page. Also look through these UE wiki pages since it does talk about Online Subsystem and my guess according to the user post on the forum you should be able to create scale down version of your own Online Subsystem.

I recommend getting into STEAM’s Online Subsystem and read this documentation

Creating your own Online SubSystem... - C++ Programming - Unreal Engine ForumsUse online subsystem for STEAM