FArchive(ing) a UTexture2D

Hi,

I’ve hit a wall while trying to write a custom NetSerialize method.
What I’m basically doing is serializing some primitive values and a texture. The primitive values are serialized with the << operator of FArchive and the texture is serialized with UTexture2D::Serialize(FArchive&). Well, I try to serialize it this way, but I hit an assert on runtime:

Bad archive positions for bulkdata.
StartPos=-1 EndPos=-1

What am I doing wrong? Did anyone successfully serialized a texture this way?

Hi! Is this texture is from Content folder or is it generated dynamically during your game? The first case can be saved by asset path, the second - by TArray (binary representation of bitmaps)

It’s a dynamic texture (I use USceneCaptureComponent2D to render the scene to a texture).
I’ll try to send it as TArray as you suggest, but I’m afraid that it’ll result in a similar error to the one I got while trying to send the texture like this

auto& textureBulkData = MyTexture->PlatformData->Mips[0].BulkData;
Ar.Serialize(textureBulkData.Lock(LOCK_READ_ONLY), textureBulkData.GetBulkDataSize());
textureBulkData.Unlock();

LogNetPartialBunch: Error: Attempted to send bunch exceeding max allowed size. BunchSize=2097167, MaximumSize=65536

The error from your last comment tells that Ar.Serialize cant work with more data than65536 bytes at once. Several steps to experiment are

  • try do the same code with very small texture and see if it work
  • if that’s true you should just serialize you BulkData by parts, for example every chunk size is 65536 except the last one.

So I tried doing this with a smaller texture and it actually worked. Then I tried to chunk it but I’m still getting the message
LogNetPartialBunch: Error: Attempted to send bunch exceeding max allowed size. BunchSize=2097167, MaximumSize=65536
as if you couldn’t serialize 1024 bytes at once in NetSerialize

Here is the code

uint8* data = (uint8*)textureMip0.BulkData.Lock(LOCK_READ_ONLY);

uint64 numOfBytesSent = 0;
uint64 numOfBytesLeftToSend = textureSize;

uint16 chunkSize = 0;
while (numOfBytesLeftToSend)
{
    if (numOfBytesLeftToSend > MAX_CHUNK_SIZE)
    {
        chunkSize = MAX_CHUNK_SIZE;
    }
    else 
    {
        chunkSize = numOfBytesLeftToSend;
    }

    Ar.Serialize(data + numOfBytesSent, chunkSize);

    numOfBytesLeftToSend -= chunkSize;
    numOfBytesSent += chunkSize;
}

textureMip0.BulkData.Unlock();

Ok, I managed to do it. Here is the code.

bool FMyStruct::NetSerialize(FArchive& Ar, class UPackageMap* Map, bool& bOutSuccess)
{
    if (Ar.IsSaving())
    {
        auto textureMip0=  MyTexture->PlatformData->Mips[0];
        
        int32 textureWidth = MyTexture->GetSizeX(); 
        int32 textureHeigth = MyTexture->GetSizeY();
        int8 textureFormat = MyTexture->GetPixelFormat();

        Ar << textureWidth;
        Ar << textureHeigth;
        Ar << textureFormat;
        
        Ar.AttachBulkData(nullptr, &textureMip0.BulkData);
    }
    else
    {
        int32 textureWidth, textureHeight;
        int8 textureFormat;

        Ar << Quality;
        Ar << textureWidth;
        Ar << textureHeight;
        Ar << textureFormat;

        
        MyTexture = UTexture2D::CreateTransient(textureWidth, textureHeight, (EPixelFormat)textureFormat);
        
        FTexture2DMipMap& textureMip0 = MyTexture->PlatformData->Mips[0];
        Ar.DetachBulkData(&textureMip0.BulkData, true);

        MyTexture->UpdateResource();
    }
    return true;
}

Two things that are missing from this snippet which I will implement next is

  1. Compress the texture/convert it to a lighter format, like JPEG

  2. Use a well-known and supported pixel format, instead of transferring it over the network.

Thank you very much for the solution! I am trying to do the same, but for some reason it seems like the Ar.AttachBulkData does not modify my FArchive at all. I assume the method might only be supported by some archives.

I’m using FBufferArchive and its size isn’t modified at all with this command, even though the textureWidth etc. do get saved in it correctly. I do check BulkData size beforehand, and the data is definitely there, it just doesn’t get attached.

Which archive type do you pass as parameter into this function in your implementation? Should it be FMemoryWriter?

I had run into an issue using the code provided by woookie2na2, as the AttachBulkData method did not do anything to the FArchvie for some reason.

I ended up using Texture->PlatformData->Mips[0].BulkData->Serialize(DataArchive, nullptr) instead of both the AttachBulkData and DetachBulkData.

I don’t know if it works for really big textures, I didn’t test that as it isn’t needed for my use case. Just wanted to mention this as an alternative solution, in case people run into the same issue.

So the full code is as follows:

bool SaveTextureToArchive(FArchive& DataArchive, UTexture2D* Texture)
{
	int32 TextureWidth = Texture->GetSizeX();
	int32 TextureHeight = Texture->GetSizeY();
	int8 TextureFormat = Texture->GetPixelFormat();
	int8 CompressionSettings = Texture->CompressionSettings;
	int8 MipGenSettings = Texture->MipGenSettings;
	uint8 IsSRGB = Texture->SRGB;
	FUntypedBulkData* BulkDataToSave = &Texture->PlatformData->Mips[0].BulkData;

	DataArchive << TextureWidth;
	DataArchive << TextureHeight;
	DataArchive << TextureFormat;
	DataArchive << CompressionSettings;
	DataArchive << MipGenSettings;
	DataArchive << IsSRGB;
	BulkDataToSave->Serialize(DataArchive, nullptr);
	
	return true;
}


UTexture2D* LoadTextureFromArchive(FArchive& DataArchive) {

	int32			TextureWidth, TextureHeight;
	int8			TextureFormat, CompressionSettings, MipGenSettings;
	uint8			IsSRGB;
	FByteBulkData*	BulkDataToFill;

	// Get texture info from archive
	DataArchive << TextureWidth;
	DataArchive << TextureHeight;
	DataArchive << TextureFormat;
	DataArchive << CompressionSettings;
	DataArchive << MipGenSettings;
	DataArchive << IsSRGB;

	// Create new texture and assign the needed settings to it
	UTexture2D* Texture = UTexture2D::CreateTransient(TextureWidth, TextureHeight, (EPixelFormat)TextureFormat);
	Texture->CompressionSettings = (TextureCompressionSettings)CompressionSettings;
	Texture->MipGenSettings = (TextureMipGenSettings)MipGenSettings;
	Texture->SRGB = IsSRGB;
	BulkDataToFill = &Texture->PlatformData->Mips[0].BulkData;
	BulkDataToFill->Serialize(DataArchive, nullptr);

	Texture->UpdateResource();
	return Texture;
}

I was using this for saving/loading files, but it can be used for network serialization too I suppose. Just in cases here is the file IO functions, which are using the above functions:

UTexture2D* LoadTextureFromFile(const FString& FullFilePath)
{
	TArray<uint8> DataLoadedFromFile;

	// Perform checks, return null if something goes wrong
	if (!(FFileHelper::LoadFileToArray(DataLoadedFromFile, *FullFilePath))) {
		DataLoadedFromFile.Empty(); 
		return nullptr; 
	}
	if (DataLoadedFromFile.Num() <= 0) return nullptr;

	FMemoryReader FromBinary = FMemoryReader(DataLoadedFromFile, true);
	FromBinary.Seek(0);
	UTexture2D* Texture = LoadTextureFromArchive(FromBinary);
	FromBinary.FlushCache();
	DataLoadedFromFile.Empty();
	FromBinary.Close();

	return Texture;
	
}

bool SaveTextureToFile(const FString& FullFilePath, UTexture2D* Texture)
{
	FBufferArchive DataToSaveToBinary;
	FMemoryWriter DataSavingArchive(DataToSaveToBinary, true);
	SaveTextureToArchive(DataSavingArchive, Texture);

	if (DataToSaveToBinary.Num() <= 0) return false;

	if (FFileHelper::SaveArrayToFile(DataToSaveToBinary, *FullFilePath)) {
		DataToSaveToBinary.FlushCache();
		DataToSaveToBinary.Empty();
		return true;
	}
	else {
		DataToSaveToBinary.FlushCache();
		DataToSaveToBinary.Empty();
		return false;
	}
}