What is the best way to replicate large amounts of data?

Development World Creation
, , , ,
1

I’m creating a game where the levels are randomly generated. I’ve been doing tests as follows:

UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Generated Mesh", ReplicatedUsing = OnRep_SetGeneratedMeshData)
FGeneratedMeshData MeshData;

(The “EditAnywhere” and “BlueprintReadWrite” flags are only used for viewing data while in game.)

GeneratedMeshData.h

USTRUCT()
struct ALPHA_API FGeneratedMeshData
{
	GENERATED_USTRUCT_BODY()

public:
	FGeneratedMeshData();
	~FGeneratedMeshData();

	UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Generated Mesh")
	FSimpleMeshData SceneProxyMeshData;
	UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Generated Mesh")
	TArray<FVector2D> SceneProxyUVData;
	UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Generated Mesh")
	FSimpleMeshData PhysicsMeshData;
};

SimpleMeshData.h

USTRUCT()
struct ALPHA_API FSimpleMeshData
{
	GENERATED_USTRUCT_BODY()

public:
	UPROPERTY(EditAnywhere, Category = "Generated Mesh")
	TArray<int32> Indices;
	UPROPERTY(EditAnywhere, Category = "Generated Mesh")
	TArray<FVector> Vertices;

	void AddVertice(FVector vertice);
	void AddTriangle(FTriangle triangle);

	int32 IndiceCount() const;
	int32 VerticeCount() const;
	int32 TriangleCount() const;

	FTriangle GetTriangleByIndex(int32 index);
	FBoxSphereBounds CalcCurrentBounds() const;
};

This seems to work rather well until my mesh data reaches around 1MB. The MeshData is only set once the mesh is complete. i.e.

DoMeshGeneration()
{
    FGeneratedMeshData md;
    [MeshGenerationStuff]
    this->MeshData = md;
}

If my replicated data is >~1MB, it crashes. My output log reads:

"Assertion failed: WriterState.CurrentChanged - OldChangedIndex == ArrayChangedCount [File:D:\BuildFarm\buildmachine_++depot+UE4-Releases+4.6\Engine\Source\Runtime\Engine\Private\RepLayout.cpp] [Line: 977]"

I can’t find anything online as far as replicating large amounts of data is concerned. Any help is very much appreciated!

2

You should be replicating the seed to your random number generator. This way you’re only replicating a small amount of data, but it gives you a bunch of info.

3

This would work great except that the mesh wont always represent what is generated by the server seed. I could also transmit the mesh in “chunks”. I’ve found ways around the issue already. Some games simply just need to send large amounts of data to the client when a player joins the game. All I want to know is the best way to do so.

4

After a bit of digging through the UE4 source code, I discovered that dynamic arrays have a defined limit for max array size of 2048 and max memory size of 1024 * 64.

\source\runtime\engine\private\replayout.cpp

void FRepLayout::SerializeProperties_DynamicArray_r( 
	FArchive &			Ar, 
	UPackageMap	*		Map,
	const int32			CmdIndex,
	uint8 *				Data,
	bool &				bHasUnmapped ) const
{
	const FRepLayoutCmd& Cmd = Cmds[ CmdIndex ];

	FScriptArray * Array = (FScriptArray *)Data;

	// Read array num
	uint16 ArrayNum = Array->Num();
	Ar << ArrayNum;

	const int MAX_ARRAY_SIZE = 2048;

	if ( ArrayNum > MAX_ARRAY_SIZE )
	{
		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum > MAX_ARRAY_SIZE (%s)" ), *Cmd.Property->GetName() );
		Ar.SetError();
		return;
	}

	const int MAX_ARRAY_MEMORY = 1024 * 64;

	if ( (int32)ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY )
	{
		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY (%s)" ), *Cmd.Property->GetName() );
		Ar.SetError();
		return;
	}

	if ( Ar.IsLoading() && ArrayNum != Array->Num() )
	{
		// If we are reading, size the array to the incoming value
		FScriptArrayHelper ArrayHelper( (UArrayProperty *)Cmd.Property, Data );
		ArrayHelper.Resize( ArrayNum );
	}

	Data = (uint8*)Array->GetData();

	for ( int32 i = 0; i < Array->Num() && !Ar.IsError(); i++ )
	{
		SerializeProperties_r( Ar, Map, CmdIndex + 1, Cmd.EndCmd - 1, Data + i * Cmd.ElementSize, bHasUnmapped );
	}
}

I tried commenting out the following lines and was able to receive larger amounts of data but not near to what I was hoping for before the client/server just stopped receiving/sending data.

const int MAX_ARRAY_SIZE = 2048;
    
    	if ( ArrayNum > MAX_ARRAY_SIZE )
    	{
    		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum > MAX_ARRAY_SIZE (%s)" ), *Cmd.Property->GetName() );
    		Ar.SetError();
    		return;
    	}
    
    	const int MAX_ARRAY_MEMORY = 1024 * 64;
    
    	if ( (int32)ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY )
    	{
    		UE_LOG( LogNetTraffic, Error, TEXT( "SerializeProperties_DynamicArray_r: ArrayNum * Cmd.ElementSize > MAX_ARRAY_MEMORY (%s)" ), *Cmd.Property->GetName() );
    		Ar.SetError();
    		return;
    	}

At the moment, I do not believe that UE4 has a network mechanism specifically dedicated to “Big Data”. While trying to find a solution to my problem, I came across this WIKI concerning TCP Sockets. Over the next day, I’m going to make a decision to either use the TCP Sockets or to customize my engine so I can tell a UFUNTION that it’s dealing with “big data” and to send the data to the client appropriately.

In conclusion, there are several answers to the question. It’s just a matter of what works best for the situation and if you are willing to do the work.

5

I’m not sure i understand you, how do you mean " the mesh wont always represent what is generated by the server seed. "

6

The mesh can be and will be updated(changed). Players can join while the game is in process. I am solving this issue by the use of tcp sockets. Eventually I am going to customize my engine and the UHT/UBT so I only have to pass “bigdata” as a specifier to a UFUNCTION.

7

Streaming large amounts of data is something we’re definitely wanting to add and get better at, but just haven’t had time to fully address.

The bandwidth caps you are seeing (and had to modify) were to protect the server from malicious clients, but we definitely need to make this optional.

As you unfortunately found out though, if you try to send larger amounts of data (usually anything above these caps), due to packet fragmentation (currently anything above 512), any amount of packet loss could put you in an infinite negative feedback loop, since any lost packet fragment will cause the entire packet to resend.

8

One thing you could also try though, is make sure this function isn’t throttling your data:

int32 UNetConnection::IsNetReady( bool Saturate )
{
	// Return whether we can send more data without saturation the connection.
	if( Saturate )
		QueuedBytes = -SendBuffer.GetNumBytes();
	return QueuedBytes+SendBuffer.GetNumBytes() <= 0;
}

Try commenting out the “if ( Saturate)” line above.

9

I did search through the engine source and commented out all data limit related code and, indeed, I was able to receive all my data but at the cost of the game “freezing” while the data was being transferred. I assume this is intended, when considering the imposed data limits, to help keep synchronicity between host and client. In the end, I decided to write my own TCP replication process.

10

I’ll have to have to check that out tonight when I get home. Thanks for the suggestion!

11

Np. Commenting out the saturate check above would definitely be temporary.

Using TCP isn’t a bad idea if that works for you if you need to stream large amounts of data reliably.

We’ll be looking into how to best deal with streaming using stock UE4 replication features at some point though.

12

I’ve decided that I will be using TCP to stream my data. The data that i need to stream is the level mesh data. It doesn’t need to be synchronized. The stream will only take place before a player joins a game in most cases. At which point I can handle other changes to the mesh with rpc replication. the rpc replication will work for most cases but just to be on the safe side, I’ll be making my checks and then if need be, make changes via TCP.

13

I was just wondering if there is now a way of dealing with large data within stock UE4 functionality yet? I’ve recently started working on a project that includes a map editor, although most of maps will likely be small you never know when someones going to be ambitious

14

I have run into the similar issue, and I think the problem could be concluded as:
Data Replicated with peek causes data lost.

After investigation, I found that UE4 used a user specified value in config file, to detect network saturation. And the algrithm is not friendly to payload with a peek which may only contain a limited data in a period time.

try add this in defaultEngine.ini

[/Script/Engine.Player]
ConfiguredLanSpeed=10000000
ConfiguredInternetSpeed=10000000

15

I Just found a patch on upstream master branch with commit bf3d7748, which make socket buffer size configurable. I think with that it can be resolved.

16

This change is scheduled for release in 4.19

17

It is now possible to set MaxRepArraySize and MaxRepArrayMemory in Project Settings under Network Settings, in case anyone is wondering.