Stream huge TArray to textfile

Development Programming & Scripting C++
1

Hi guys!

I currently work on a function that exports point cloud data (stored in an array) to a textfile. I currently use the** “SaveStringArrayToFile”** function, but with to many points the programm crashes.

I assume that this function is not streaming the data, thus flooding my RAM (I get out of memory everywhere and my laptop freezes until Unreal stops the program by itself).

Is there a simple way like in Python, where the data gets automatically streamed to avoid this problem?

My current function:



bool UPointCloud::BP_ExportCloud(FString SaveDirectory, FString FileName, bool AllowOverWriting = false, bool ExportSelections = false)
{
    //Set complete file path
    SaveDirectory += "\\";
    SaveDirectory += FileName;

    //Abort if File already exists and overwriting not allowed
    if (!AllowOverWriting)
    {
        if (FPlatformFileManager::Get().GetPlatformFile().FileExists(*SaveDirectory))
        {
            return false;
        }
    }

    //Build Final String
    TArray<FString> FinalString;
    FString Line = "";
    FString Delimiter = ", ";
    for (auto const& point : Points)
    {
        Line += FString::SanitizeFloat(point.Location.X) + Delimiter;
        Line += FString::SanitizeFloat(point.Location.Y) + Delimiter;
        Line += FString::SanitizeFloat(point.Location.Z) + Delimiter;
        Line += FString::SanitizeFloat(point.Color.R) + Delimiter;
        Line += FString::SanitizeFloat(point.Color.G) + Delimiter;
        Line += FString::SanitizeFloat(point.Color.B) + LINE_TERMINATOR;
        FinalString.Add(Line); 
    }
    return FFileHelper::SaveStringArrayToFile(FinalString, *SaveDirectory);
}

Just wanted to add that I am quite a beginner with C++ and Unreal, since I started just two months ago.

Thanks everyone for any tipps in the right direction!!

2

Why do you need it to be a text file?
Those string append operations make it super slow and generating garbage memory, plus freezing game thread.

3

Thanks for your reply! I do need some way of further processing the resulting point cloud, a textfile is the simplest format that came in to my mind.

Until now I use the standard C++ ofstream and iostream tools to get the job done. Not sure what drawbacks that might have, but well, it works :smiley:

4

In that case I think you could cut out the FinalString container to avoid allocating unnecessary memory while reading the points buffer, if you didn’t do that already :slight_smile:

5

You say you’re currently using SaveStringArrayToFile to start, then in the next post, you say you’re using iostream, and your code says you’re using SaveStringArrayToFile. I’m so confused.

You probably want to use FString::Printf FString | Unreal Engine Documentation

in a single statement, rather than adding tons of strings together in individual statements.

And no, this doesn’t have anything to do with streaming. You’re doing a very time consuming operation on the main thread. Writing huge files is not something you should do on the game thread. I don’t have any specific advice as to how to offload it to a secondary thread, because I’ve never needed to do that before.

6

Are you doing point cloud visualization or surface reconstruction?

I think your problem here is a small typo in your code. The FString Line should be inside the for loop! The way it is now, you are writing everything you previously wrote to each line, like a pyramid.

Your final file would contain (Point.Num() * Point.Num())/2 points!



bool UPointCloud::BP_ExportCloud(FString SaveDirectory, FString FileName, bool AllowOverWriting = false, bool ExportSelections = false)
{
    //Set complete file path
    SaveDirectory += "\\";
    SaveDirectory += FileName;

    //Abort if File already exists and overwriting not allowed
    if (!AllowOverWriting)
    {
        if (FPlatformFileManager::Get().GetPlatformFile().FileExists(*SaveDirectory))
        {
            return false;
        }
    }

    //Build Final String
    TArray<FString> FinalString;
    FString Delimiter = ", ";
    for (auto const& point : Points)
    {
        FString Line = ""; // need a fresh FString on each pass through the loop

        Line += FString::SanitizeFloat(point.Location.X) + Delimiter;
        Line += FString::SanitizeFloat(point.Location.Y) + Delimiter;
        Line += FString::SanitizeFloat(point.Location.Z) + Delimiter;
        Line += FString::SanitizeFloat(point.Color.R) + Delimiter;
        Line += FString::SanitizeFloat(point.Color.G) + Delimiter;
        Line += FString::SanitizeFloat(point.Color.B) + LINE_TERMINATOR;
        FinalString.Add(Line);
    }
    return FFileHelper::SaveStringArrayToFile(FinalString, *SaveDirectory);
}
7

Glad to know some one here working with point cloud as well.

In this case, SaveStringArrayToFile may not suit for handling huge size of array, since that is 32 bits version and be limited by max length of FString which is int32.

Hope the code like below could help.



#include "FileManager.h"
void UMyBlueprintFunctionLibrary::WriteExcerptDataIntoFile(FString FileSource, TArray<FVector> InExcerptData)
{
    FText CheckoutFailReason;
    bool bNewFile = true;
    bool bCheckoutOrAddSucceeded = true;
    if (FPaths::FileExists(FileSource))
    {
        // Check out the existing file
        bNewFile = false;
    }

    FArchive* FileWriter = IFileManager::Get().CreateFileWriter(*FileSource, EFileWrite::FILEWRITE_Append | EFileWrite::FILEWRITE_AllowRead | EFileWrite::FILEWRITE_EvenIfReadOnly);

    if (bNewFile)
    {
        FString FileHeader;
        FileHeader += "Start Recording:";
        FileHeader += LINE_TERMINATOR;

        FileWriter->Serialize(TCHAR_TO_ANSI(*FileHeader), FileHeader.Len());
    }
    else
    {
        FileWriter->Seek(FMath::Max(FileWriter->TotalSize(), (int64)0));
    }

    int TotalNum = InExcerptData.Num();
    int Count = 0;
    int StepNum = 100;
    while (Count < TotalNum)
    {
        FString NewExcerpt;
        for (int i = 0; i < StepNum; i++)
        {
            FVector& Vec = InExcerptData[Count];            
            NewExcerpt += FString::Printf(TEXT("Vec:%f,%f,%f
"), Vec.X, Vec.Y, Vec.Z);
            Count++;
        }
        FileWriter->Serialize(TCHAR_TO_ANSI(*NewExcerpt), NewExcerpt.Len());
    }

    FileWriter->Close();
    delete FileWriter;
    return;
}
8

Nice to meet someone, that knows the struggle :smiley:

Thank you, I will give that a test run later!