Download

Tech Note: The FForkProcess Class for Managing Forking Dedicated Servers in Linux

Apr 7, 2021.Knowledge

Description:
This article is meant to supplement the Server Hosting Advice article you can find here.
Mentioned in that article is how a Linux dedicated server can fork child processes from a master process in order to share allocated memory. This functionality was recently exposed to the engine in 4.26, but one class relevant to this process that was not added is FForkProcess. This class manages forking the dedicated server with ForkIfRequested(), which can be called from the game instance once the dedicated server has completed its startup.

Potential Impact:
Limited: Users attempting to implement forking within their dedicated servers do not have access to FForkProcess.

Solution:
FForkProcess is planned to be migrated to UE. In the meanwhile, you can find the code for this class below, with some sections containing sensitive information removed.

ForkProcess.h

// Copyright Epic Games, Inc. All Rights Reserved.

#pragma once

#include "Delegates/IDelegateInstance.h"

class UWorld;

/** Class to manage forking dedicated servers in Linux */
class FForkProcess
{
public:
	/**
	* This is a function that halts execution and waits for signals to cause forked processes to be created and continue execution.
	* This forking behavior is only allowed on linux servers, but note that this function needs to get called even if you are not forking so perf counters properly get created.
	* Forking also is only allowed when running with -nothreading, and only happens is -WaitAndFork is specified on the command line.
	* It will fork child processes that use the shared memory from this process.
	* This will only work for the parent process, child processes cannot fork further.
	* The parent process will return when GIsRequestingExit is true. SIGRTMIN+1 is used to cause a fork to happen.
	* If sigqueue is used, the payload int will be split into the upper and lower uint16 values. The upper value is a "cookie" and the
	*     lower value is an "index". These two values will be used to name the process using the pattern DS-<cookie>-<index>. This name
	*     can be used to uniquely discover the process that was spawned.
	* If -NumForks=x is suppled on the command line, x forks will be made when the function is called.
	* If -WaitAndForkCmdLinePath=Foo is suppled, the command line parameters of the child processes will be filled out with the contents
	*     of files found in the directory referred to by Foo, where the child's "index" is the name of the file to be read in the directory.
	* If -WaitAndForkRequireResponse is on the command line, child processes will not proceed after being spawned until a SIGRTMIN+2 signal is sent to them.
	* A world is required to shut down the current net driver and to get the game instance to set up perf counters.
	*/
	static void ForkIfRequested(UWorld* World);

	/** Should the process exit when the last player has left or should the process reload and create a new session */
	static bool ShutdownProcessAtSessionEnd();


	/** Set up this process to ask the parent process to shut down when this one exits */
	static void MarkShouldCloseParentProcessWhenShuttingDown();

public:

	/** Handle used to remove the OnEndFrame delegate after it is triggered once */
	static FDelegateHandle OnEndFrameHandle;

	/** Delegate called immediately after forking */
	static void OnEndFramePostFork();

private:
	/** Only attempt to fork exactly once. Processes should not attempt to fork again after the first request, child or parent. */
	static bool bForkAlreadyAttempted;

	/** If true, when this process exists, it will return an exit code to indicate that the parent process should also shut down */
	static bool bShutdownWithExitCodeToCloseParent;
};

ForkProcess.cpp

// Copyright Epic Games, Inc. All Rights Reserved.

#include "Fork/ForkProcess.h"
#include "OnlineSubsystemUtils.h"
#include "IPlatformFilePak.h"
#include "HAL/PlatformFilemanager.h"
#include "HAL/FileManager.h"
#include "HAL/ThreadHeartBeat.h"
#include "HAL/IConsoleManager.h"
#include "HAL/PlatformMemory.h"
#include "UnrealEngine.h"
#include "Misc/CoreDelegates.h"
#include "Misc/Fork.h"
#include "GameInstance.h"
#include "HttpModule.h"
#include "HttpManager.h"
#include "NetworkReplayStreaming.h"
#include "HAL/ThreadManager.h"
#include "ProfilingDebugging/CsvProfiler.h"

bool FForkProcess::bForkAlreadyAttempted = false;
bool FForkProcess::bShutdownWithExitCodeToCloseParent = false;

FDelegateHandle FForkProcess::OnEndFrameHandle;

namespace FForkProcessInternal
{
	static bool CVar_Server_MultithreadTaskGraph = true;

	static bool CVar_Server_ReloadPakReaders = true;

	static FDelegateHandle OnHotfixTaskGraph;

	/** Delegate for resetting the task graph threading strategy */
	void OnEndFrameHotfixTaskGraph();

	/**
	 * Are we doing a real fork and generating child processes from the master who received the -WaitAndFork commandline.
	 * Will be true on child processes even if they do not receive the WaitAndFork command.
	 */
	bool IsRealForkRequested()
	{
		// We cache the value since only the master process will receive WaitAndFork
		static const bool bRealForkRequested = FParse::Param(FCommandLine::Get(), TEXT("WaitAndFork"));
		return bRealForkRequested;
	}

	/**
	* Fake Forking is when we run the Fork codepath without actually duplicating the process.
	* Can be used on platforms that do not support forking or to simply debug the fork behavior without attaching to the new process.
	* Note: The master process is considered a child process after the Fork event.
	*/
	bool IsFakeForking()
        {
#if UE_SERVER
        	const bool bRealForkRequested = FForkProcessInternal::IsRealForkRequested();
        	if (bRealForkRequested)
        	{
        		return false;
        	}

        	const bool bNoFakeForking = FParse::Param(FCommandLine::Get(), TEXT("NoFakeForking"));
        	if (bNoFakeForking)
        	{
        		return false;
        	}

        	// Default to have dedicated servers simulate the fork since it's closer to the live environment.
        	return true;
#else
	        return false;
#endif
        }
}

#ifndef WAIT_AND_FORK_PARENT_SHUTDOWN_EXIT_CODE
	#define WAIT_AND_FORK_PARENT_SHUTDOWN_EXIT_CODE 0
#endif


void FForkProcess::ForkIfRequested(UWorld* World)
{
	// Only attempt to fork exactly once. Child instances should not fork. This is an important protection against accidental fork bombs.
	if (bForkAlreadyAttempted)
	{
		return;
	}

	bForkAlreadyAttempted = true;

	check(World);

	// Should this process be duplicated and create child processes
	const bool bRealFork = FForkProcessInternal::IsRealForkRequested();

	// Should we run the Fork codepath without duplicating the process.
	const bool bFakeFork = !bRealFork && FForkProcessInternal::IsFakeForking();

	if (bRealFork || bFakeFork)
	{
		// Do we turn off and restart the GameNetDriver after the fork
		const bool bRestartNetDriver = bRealFork;
		if (bRestartNetDriver)
		{
			GEngine->ShutdownWorldNetDriver(World);
		}

		// Flush the http manager to make sure there are no outstanding http requests when we fork
		FHttpManager& HttpManager = FHttpModule::Get().GetHttpManager();
		HttpManager.OnBeforeFork();

		// Do we reset the pak readers on child processes
		const bool bResetPakReaders = bRealFork;

		TArray<FString> AllPakFiles;
		TArray<FString> PakFolders;
		FPakPlatformFile* PakPlatformFile = nullptr;

		if (bResetPakReaders)
		{
			PakPlatformFile = (FPakPlatformFile*)(FPlatformFileManager::Get().FindPlatformFile(FPakPlatformFile::GetTypeName()));
			if (PakPlatformFile == nullptr)
			{
				UE_LOG(LogEngine, Fatal, TEXT("You cannot use WaitAndFork without using pak files."));
			}

			PakPlatformFile->GetPakFolders(FCommandLine::Get(), PakFolders);
			for (const FString& PakFolder : PakFolders)
			{
				TArray<FString> PakFiles;
				IFileManager::Get().FindFiles(PakFiles, *PakFolder, TEXT(".pak"));
				for (const FString& PakFile : PakFiles)
				{
					AllPakFiles.Add(PakFolder / PakFile);
				}
			}

			if (AllPakFiles.Num() == 0)
			{
				UE_LOG(LogEngine, Fatal, TEXT("WaitAndFork cannot proceed without finding pak files."));
			}

			if (!FForkProcessInternal::CVar_Server_ReloadPakReaders)
			{
				for (const FString& PakFile : AllPakFiles)
				{
					if (!FCoreDelegates::OnUnmountPak.Execute(PakFile))
					{
						UE_LOG(LogEngine, Fatal, TEXT("WaitAndFork failed to unmount pak %s."), *PakFile);
					}
				}
			}
		}

		// ******** The fork happens here! ********
		FPlatformProcess::EWaitAndForkResult Result = FPlatformProcess::EWaitAndForkResult::Error;
		if (bRealFork)
		{
			Result = FPlatformProcess::WaitAndFork();
		}
		else if (bFakeFork)
		{
			// The master process becomes a child when fake forking.
			Result = FPlatformProcess::EWaitAndForkResult::Child;
			FForkProcessHelper::SetIsForkedChildProcess();
		}
		// ******** The fork happened! This is now either the parent process exiting gracefully (or erroring) or the new child process starting up ********

		if (bResetPakReaders && !FForkProcessInternal::CVar_Server_ReloadPakReaders)
		{
			for (const FString& PakFile : AllPakFiles)
			{
				if (!FCoreDelegates::MountPak.Execute(PakFile, 0))
				{
					UE_LOG(LogEngine, Fatal, TEXT("WaitAndFork failed to mount pak %s."), *PakFile);
				}
			}
		}

		// Restart the http manager
		HttpManager.OnAfterFork();

		if (Result == FPlatformProcess::EWaitAndForkResult::Error)
		{
			UE_LOG(LogEngine, Fatal, TEXT("There was an error attempting to fork. Are you sure you launched with -nothreading?"));
		}
		else if (Result == FPlatformProcess::EWaitAndForkResult::Parent)
		{
			// The parent exited gracefully
		}
		else if (Result == FPlatformProcess::EWaitAndForkResult::Child)
		{
			if (bResetPakReaders && FForkProcessInternal::CVar_Server_ReloadPakReaders)
			{
				PakPlatformFile->ReloadPakReaders();
			}

			FGameThreadHitchHeartBeat::Get().Restart();

			FURL::StaticExit();
			FURL::StaticInit();

			// Here we do some post-fork setup for the game instance. 

			OnEndFrameHandle = FCoreDelegates::OnEndFrame.AddStatic(FForkProcess::OnEndFramePostFork);

                        // re-init QOS
                        FQosInterface::Get()->Init();

			// Here we reinitialize the IOnlineTitleFile dispatcher

			if (bRestartNetDriver)
			{
				FURL DefaultURL;
				World->Listen(DefaultURL);
			}

			// We don't want child processes to write to all the static memory we are sharing during shutdown, so we will exit before the
			// UObject system is torn down. For this reason, BeginDestroy/destructors for these objects will not be called for child processes.
			if (bRealFork)
			{
				FCoreDelegates::OnPreExit.AddStatic([]()
				{
					UE_LOG(LogEngine, Display, TEXT("Forked child process flushing http requests before exiting."));
					FHttpManager& LocalHttpManager = FHttpModule::Get().GetHttpManager();
					LocalHttpManager.Flush(true);

					uint8 ExitCode = bShutdownWithExitCodeToCloseParent ? WAIT_AND_FORK_PARENT_SHUTDOWN_EXIT_CODE : 0;
					UE_LOG(LogEngine, Display, TEXT("Forked child process exiting cleanly with exit code %d."), ExitCode);
					if (GLog)
					{
						GLog->Flush();
					}

					FPlatformMisc::RequestExitWithStatus(true, ExitCode);
				});
			}
		}
		else
		{
			UE_LOG(LogEngine, Fatal, TEXT("Unknown return value from WaitAndFork. Result:%d"), (uint8)Result);
		}
	}

    // Here we initialize perf counters for the dedicated server game instance
}

void FForkProcess::MarkShouldCloseParentProcessWhenShuttingDown()
{
	// Only forked children of a process may request to shutdown a parent process
	if (FForkProcessHelper::IsForkedChildProcess())
	{
		bShutdownWithExitCodeToCloseParent = true;
	}
}

bool FForkProcess::ShutdownProcessAtSessionEnd()
{
	return FForkProcessHelper::IsForkedChildProcess() && !FForkProcessInternal::IsFakeForking();
}

void FForkProcess::OnEndFramePostFork()
{
	// This delegate is only called once
	FCoreDelegates::OnEndFrame.Remove(OnEndFrameHandle);
	OnEndFrameHandle.Reset();

	if( FForkProcessHelper::SupportsMultithreadingPostFork() )
	{
		// Convert forkable threads into real threads
		FForkProcessHelper::OnForkingOccured();

		if( FForkProcessInternal::CVar_Server_MultithreadTaskGraph )
		{
			FTaskGraphInterface::Shutdown();	
			FTaskGraphInterface::Startup(0);
			FTaskGraphInterface::Get().AttachToThread(ENamedThreads::GameThread);
		}
	}

	//TODO: Add a CoreDelegate::OnEndFramePostFork so systems can hook into it instead of us telling them we forked.

	FHttpModule::Get().GetHttpManager().OnEndFramePostFork();

#if CSV_PROFILER
	FCsvProfiler::Get()->OnEndFramePostFork();
#endif
}

/** This CVar is used to flip back to unmounting the pak files before we fork and having the children re-mount
  * these pak files. This was causing a hit to USS so a new way is to simply allow the children to reload the
  * pak readers which have a handle on the low level file handle
  */
static FAutoConsoleVariableRef CVarServerReloadPakReaders(
	TEXT("Server.ReloadPakReaders"),
	FForkProcessInternal::CVar_Server_ReloadPakReaders,
	TEXT("Has child process re-create the pak readers over unmounting then remounting pak files"),
	ECVF_Default);