Announcement

Collapse
No announcement yet.

Stream huge TArray to textfile

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Stream huge TArray to textfile

    Hi guys!

    I currently work on a function that exports point cloud data (stored in an array) to a textfile. I currently use the "SaveStringArrayToFile" function, but with to many points the programm crashes.

    I assume that this function is not streaming the data, thus flooding my RAM (I get out of memory everywhere and my laptop freezes until Unreal stops the program by itself).

    Is there a simple way like in Python, where the data gets automatically streamed to avoid this problem?

    My current function:

    Code:
    bool UPointCloud::BP_ExportCloud(FString SaveDirectory, FString FileName, bool AllowOverWriting = false, bool ExportSelections = false)
    {
        //Set complete file path
        SaveDirectory += "\\";
        SaveDirectory += FileName;
    
        //Abort if File already exists and overwriting not allowed
        if (!AllowOverWriting)
        {
            if (FPlatformFileManager::Get().GetPlatformFile().FileExists(*SaveDirectory))
            {
                return false;
            }
        }
    
        //Build Final String
        TArray<FString> FinalString;
        FString Line = "";
        FString Delimiter = ", ";
        for (auto const& point : Points)
        {
            Line += FString::SanitizeFloat(point.Location.X) + Delimiter;
            Line += FString::SanitizeFloat(point.Location.Y) + Delimiter;
            Line += FString::SanitizeFloat(point.Location.Z) + Delimiter;
            Line += FString::SanitizeFloat(point.Color.R) + Delimiter;
            Line += FString::SanitizeFloat(point.Color.G) + Delimiter;
            Line += FString::SanitizeFloat(point.Color.B) + LINE_TERMINATOR;
            FinalString.Add(Line); 
        }
        return FFileHelper::SaveStringArrayToFile(FinalString, *SaveDirectory);
    }
    Just wanted to add that I am quite a beginner with C++ and Unreal, since I started just two months ago.

    Thanks everyone for any tipps in the right direction!!


    #2
    Why do you need it to be a text file?
    Those string append operations make it super slow and generating garbage memory, plus freezing game thread.
    | Finite State Machine | Savior | USQLite | Object-Pool | Sound-Occlusion | Property Transfer | Magic Nodes | MORE |

    Comment


      #3
      Originally posted by BrUnO XaVIeR View Post
      Why do you need it to be a text file?
      Those string append operations make it super slow and generating garbage memory, plus freezing game thread.
      Thanks for your reply! I do need some way of further processing the resulting point cloud, a textfile is the simplest format that came in to my mind.

      Until now I use the standard C++ ofstream and iostream tools to get the job done. Not sure what drawbacks that might have, but well, it works

      Comment


        #4
        In that case I think you could cut out the FinalString container to avoid allocating unnecessary memory while reading the points buffer, if you didn't do that already
        | Finite State Machine | Savior | USQLite | Object-Pool | Sound-Occlusion | Property Transfer | Magic Nodes | MORE |

        Comment


          #5
          You say you're currently using SaveStringArrayToFile to start, then in the next post, you say you're using iostream, and your code says you're using SaveStringArrayToFile. I'm so confused.

          You probably want to use FString::Printf https://docs.unrealengine.com/en-us/...ndling/FString

          in a single statement, rather than adding tons of strings together in individual statements.

          And no, this doesn't have anything to do with streaming. You're doing a very time consuming operation on the main thread. Writing huge files is not something you should do on the game thread. I don't have any specific advice as to how to offload it to a secondary thread, because I've never needed to do that before.



          Comment


            #6
            Are you doing point cloud visualization or surface reconstruction?

            I think your problem here is a small typo in your code. The FString Line should be inside the for loop! The way it is now, you are writing everything you previously wrote to each line, like a pyramid.

            Your final file would contain (Point.Num() * Point.Num())/2 points!

            Code:
            bool UPointCloud::BP_ExportCloud(FString SaveDirectory, FString FileName, bool AllowOverWriting = false, bool ExportSelections = false)
            {
                //Set complete file path
                SaveDirectory += "\\";
                SaveDirectory += FileName;
            
                //Abort if File already exists and overwriting not allowed
                if (!AllowOverWriting)
                {
                    if (FPlatformFileManager::Get().GetPlatformFile().FileExists(*SaveDirectory))
                    {
                        return false;
                    }
                }
            
                //Build Final String
                TArray<FString> FinalString;
                FString Delimiter = ", ";
                for (auto const& point : Points)
                {
                    FString Line = ""; // need a fresh FString on each pass through the loop
            
                    Line += FString::SanitizeFloat(point.Location.X) + Delimiter;
                    Line += FString::SanitizeFloat(point.Location.Y) + Delimiter;
                    Line += FString::SanitizeFloat(point.Location.Z) + Delimiter;
                    Line += FString::SanitizeFloat(point.Color.R) + Delimiter;
                    Line += FString::SanitizeFloat(point.Color.G) + Delimiter;
                    Line += FString::SanitizeFloat(point.Color.B) + LINE_TERMINATOR;
                    FinalString.Add(Line);
                }
                return FFileHelper::SaveStringArrayToFile(FinalString, *SaveDirectory);
            }
            Last edited by timtimmy; 05-21-2019, 08:32 PM.

            Comment


              #7
              Glad to know some one here working with point cloud as well.

              In this case, SaveStringArrayToFile may not suit for handling huge size of array, since that is 32 bits version and be limited by max length of FString which is int32.

              Hope the code like below could help.

              Code:
              #include "FileManager.h"
              void UMyBlueprintFunctionLibrary::WriteExcerptDataIntoFile(FString FileSource, TArray<FVector> InExcerptData)
              {
                  FText CheckoutFailReason;
                  bool bNewFile = true;
                  bool bCheckoutOrAddSucceeded = true;
                  if (FPaths::FileExists(FileSource))
                  {
                      // Check out the existing file
                      bNewFile = false;
                  }
              
                  FArchive* FileWriter = IFileManager::Get().CreateFileWriter(*FileSource, EFileWrite::FILEWRITE_Append | EFileWrite::FILEWRITE_AllowRead | EFileWrite::FILEWRITE_EvenIfReadOnly);
              
                  if (bNewFile)
                  {
                      FString FileHeader;
                      FileHeader += "Start Recording:";
                      FileHeader += LINE_TERMINATOR;
              
                      FileWriter->Serialize(TCHAR_TO_ANSI(*FileHeader), FileHeader.Len());
                  }
                  else
                  {
                      FileWriter->Seek(FMath::Max(FileWriter->TotalSize(), (int64)0));
                  }
              
                  int TotalNum = InExcerptData.Num();
                  int Count = 0;
                  int StepNum = 100;
                  while (Count < TotalNum)
                  {
                      FString NewExcerpt;
                      for (int i = 0; i < StepNum; i++)
                      {
                          FVector& Vec = InExcerptData[Count];            
                          NewExcerpt += FString::Printf(TEXT("Vec:%f,%f,%f\r\n"), Vec.X, Vec.Y, Vec.Z);
                          Count++;
                      }
                      FileWriter->Serialize(TCHAR_TO_ANSI(*NewExcerpt), NewExcerpt.Len());
                  }
              
                  FileWriter->Close();
                  delete FileWriter;
                  return;
              }

              Comment


                #8
                Originally posted by henryLiu View Post
                Glad to know some one here working with point cloud as well.
                Nice to meet someone, that knows the struggle

                Originally posted by henryLiu View Post

                In this case, SaveStringArrayToFile may not suit for handling huge size of array, since that is 32 bits version and be limited by max length of FString which is int32.

                Hope the code like below could help.

                Code:
                #include "FileManager.h"
                void UMyBlueprintFunctionLibrary::WriteExcerptDataIntoFile(FString FileSource, TArray<FVector> InExcerptData)
                {
                FText CheckoutFailReason;
                bool bNewFile = true;
                bool bCheckoutOrAddSucceeded = true;
                if (FPaths::FileExists(FileSource))
                {
                // Check out the existing file
                bNewFile = false;
                }
                
                FArchive* FileWriter = IFileManager::Get().CreateFileWriter(*FileSource, EFileWrite::FILEWRITE_Append | EFileWrite::FILEWRITE_AllowRead | EFileWrite::FILEWRITE_EvenIfReadOnly);
                
                if (bNewFile)
                {
                FString FileHeader;
                FileHeader += "Start Recording:";
                FileHeader += LINE_TERMINATOR;
                
                FileWriter->Serialize(TCHAR_TO_ANSI(*FileHeader), FileHeader.Len());
                }
                else
                {
                FileWriter->Seek(FMath::Max(FileWriter->TotalSize(), (int64)0));
                }
                
                int TotalNum = InExcerptData.Num();
                int Count = 0;
                int StepNum = 100;
                while (Count < TotalNum)
                {
                FString NewExcerpt;
                for (int i = 0; i < StepNum; i++)
                {
                FVector& Vec = InExcerptData[Count];
                NewExcerpt += FString::Printf(TEXT("Vec:%f,%f,%f\r\n"), Vec.X, Vec.Y, Vec.Z);
                Count++;
                }
                FileWriter->Serialize(TCHAR_TO_ANSI(*NewExcerpt), NewExcerpt.Len());
                }
                
                FileWriter->Close();
                delete FileWriter;
                return;
                }
                Thank you, I will give that a test run later!

                Comment

                Working...
                X