What is the best way to replicate large amounts of data?

[Mattlauk]

After a bit of digging through the UE4 source code, I discovered that dynamic arrays have a defined limit for max array size of 2048 and max memory size of 1024 * 64.

\source\runtime\engine\private\replayout.cpp

void FRepLayout::SerializeProperties_DynamicArray_r( 
	FArchive &			Ar, 
	UPackageMap	*		Map,
	const int32			CmdIndex,
	uint8 *				Data,
	bool &				bHasUnmapped ) const
{
	const FRepLayoutCmd& Cmd = Cmds[ CmdIndex ];

	FScriptArray * Array = (FScriptArray *)Data;

	// Read array num
	uint16 ArrayNum = Array->Num();
	Ar << ArrayNum;

	const int MAX_ARRAY_SIZE = 2048;

	if ( ArrayNum > MAX_ARRAY_SIZE )
	{
		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum > MAX_ARRAY_SIZE (%s)" ), *Cmd.Property->GetName() );
		Ar.SetError();
		return;
	}

	const int MAX_ARRAY_MEMORY = 1024 * 64;

	if ( (int32)ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY )
	{
		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY (%s)" ), *Cmd.Property->GetName() );
		Ar.SetError();
		return;
	}

	if ( Ar.IsLoading() && ArrayNum != Array->Num() )
	{
		// If we are reading, size the array to the incoming value
		FScriptArrayHelper ArrayHelper( (UArrayProperty *)Cmd.Property, Data );
		ArrayHelper.Resize( ArrayNum );
	}

	Data = (uint8*)Array->GetData();

	for ( int32 i = 0; i < Array->Num() && !Ar.IsError(); i++ )
	{
		SerializeProperties_r( Ar, Map, CmdIndex + 1, Cmd.EndCmd - 1, Data + i * Cmd.ElementSize, bHasUnmapped );
	}
}

I tried commenting out the following lines and was able to receive larger amounts of data but not near to what I was hoping for before the client/server just stopped receiving/sending data.

const int MAX_ARRAY_SIZE = 2048;
    
    	if ( ArrayNum > MAX_ARRAY_SIZE )
    	{
    		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum > MAX_ARRAY_SIZE (%s)" ), *Cmd.Property->GetName() );
    		Ar.SetError();
    		return;
    	}
    
    	const int MAX_ARRAY_MEMORY = 1024 * 64;
    
    	if ( (int32)ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY )
    	{
    		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY (%s)" ), *Cmd.Property->GetName() );
    		Ar.SetError();
    		return;
    	}

At the moment, I do not believe that UE4 has a network mechanism specifically dedicated to “Big Data”. While trying to find a solution to my problem, I came across this WIKI concerning TCP Sockets. Over the next day, I’m going to make a decision to either use the TCP Sockets or to customize my engine so I can tell a UFUNTION that it’s dealing with “big data” and to send the data to the client appropriately.

In conclusion, there are several answers to the question. It’s just a matter of what works best for the situation and if you are willing to do the work.