Announcement

Collapse
No announcement yet.

Why can I not add a struct to a TArray inside an async task?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Why can I not add a struct to a TArray inside an async task?


    Can anyone answer this question for me?

    With a struct like this:

    Code:
    struct FMyFloatArrayStruct
    {
    public:
        TArray<float> FloatArray;
    };
    A defined array in the actors header like this:

    Code:
    TArray<FMyFloatArrayStruct> TestStructArray;
    An actor Destroyed() function like this:

    Code:
    void AMyActor:Destroyed()
    {
        TestStructArray.Empty();
    }
    Running this code on the actors MAIN THREAD clears out all of the RAM when the actor is deleted in the level editor window:

    Code:
    TestStructArray.Empty();
    TestStructArray.Init(FMyFloatArrayStruct(), 300000000);
    
    for (int32 Index = 0; Index < 300000000; Index++)
    {
        FMyFloatArrayStruct* Struct1 = &TestStructArray[Index];
        Struct1->FloatArray.Push(FMath::Rand());
        Struct1 = nullptr;
    }
    ...but this code run using an ASYNC TASK, does NOT clear out all of its RAM when the actor is deleted in the level editor window:

    Code:
    AsyncTask(ENamedThreads::AnyBackgroundThreadNormalTask, [&]()
    {
       TestStructArray.Empty();
       TestStructArray.Init(FMyFloatArrayStruct(), 300000000);
    
       for (int32 Index = 0; Index < 300000000; Index++)
       {
          FMyFloatArrayStruct* Struct1 = &TestStructArray[Index];
          Struct1->FloatArray.Push(FMath::Rand());
          Struct1 = nullptr;
       }
    });
    My RAM for both functions goes up to 9800Mb when fully finished.

    The RAM for the MAIN THREAD version goes back down to 600Mb after the actor is deleted.
    The RAM for the ASYNC TASK version goes down to 5200Mb after the actor is deleted.

    It would seem with this code that I cannot add any structs to the array within an async task.

    Why is there a difference?

    #2
    Is this not a problem for anyone else?

    I have tried:

    FRunnable
    FNonAbandonableTask
    AsyncTask
    Future/Async

    Thread
    ThreadPool
    TaskGraph

    FCriticalSection with FScopeLock to lock the array adding/pushing section.

    Nothing works... if trying to add a struct to an array and then have all the memory returned when the actor is deleted.

    If I create an array of floats it works fine but not with an array of structs which contains an array of floats.

    Is this even a memory leak or is it supposed to work this way?

    I have found many examples of async code including looking at the UE4 source code but it all just fills the RAM and only partially releases. Never back to what it started with.

    Please, anyone?

    I just need an example that works.
    Last edited by 3dev; 10-25-2020, 10:02 AM.

    Comment


      #3
      Have you tried it on Shipping build? Because it fully frees up memory for me on Shipping build so I wouldn't worry about it too much especially that on non-shipping build when executing it multiple times the memory is reused and doesn't add up so it will be harder for you to run out of memory while developing.

      If you however need to have it freed on non-shipping builds for some reason, I found workaround, memory gets freed properly when dynamically allocating TArray<FMyFloatArrayStruct> and then freeing it on e.g. EndPlay but from within AsyncTask(ENamedThreads::AnyBackgroundThreadNormalTask...)

      Comment


        #4
        Thank you for answering that.

        I tried creating an async task using:

        Code:
        Async(EAsyncExecution::TaskGraph, [this]()
        {
            for (int32 Index = 0; Index < 300000000; Index++)
            {
                FMyFloatArrayStruct struct1;
                struct1.FloatArray.Add(0.f);
                struct1.FloatArray.Add(1.f);
                TestStructArray.Push(struct1);
            }
        });
        Which adds all the structs to the array correctly.

        Then in the Destroyed() function of the actor I used:

        Code:
        Async(EAsyncExecution::TaskGraph, [this]()
        {
            TestStructArray.Empty();
        }
        Which empty's the array and returns the memory successfully.

        If you just run TestStructArray.Empty() without it being inside an async task then it does not return all of the memory.

        If you don't fill the array in an async task and fill it on the main thread then running Empty() on the array does work when also run on the main thread.

        Can anyone explain why we cannot fill the array in the async task and then empty it in the main thread?

        The question also is how to fill the array in an FRunnable thread then empty it using some other thread? This I cannot get working.
        Last edited by 3dev; 10-29-2020, 12:07 PM.

        Comment


          #5
          Yeah i would like to know the answer to this also. I am trying to put 3 TArrays into 3 different structs that are all one huge structure i cant get it to reconize them in the structs, unless i do not use TArray.
          Code:
           struct WeaponStatsData// Structure for each guns data
          {
          //IS THIS NOT WORKING BECAUSE OF THE DIFFERENT TYPES BELOW?
          char weaponName[25];
          char playTimeString[25];
          int ammo = 0;
          int tags = 0;
          int tagged = 0;
          int shots = 0;
          int batteryLife = 0;
          float co2 = 0.0;
          float timeUsed = 0.0;
          };
          
          struct LoadoutStatsData// Structure for loadout settings
          {
          //WILL CONTAIN INT32 AND CHARS AND FLOATS AS ABOVE
          TArray <struct WeaponStatsData> weaponsInfo;
          };
          
          struct ProfileStatsData //Structure for profile settings
          {
          //WILL CONTAIN INT32 AND CHARS AND FLOATS AS ABOVE
          TArray<struct LoadoutStatsData>loadOutInfo;
          
          };
          TArray<struct ProfileStatsData>profileInfo;

          If i make them like this i can get them to work.
          Code:
          ProfileStatsData profileInfo[3];
          But then i can not use the .Num() to get its elements to use in fors. or the .SetNum(yourNumOfElements); To set how many elements you want it to be.
          Anyone with experience with the TArray is this possible in structures and if so, point us in the right direction. thanks for reading.

          Comment


            #6
            What you are creating there is a standard fixed size array that contains 4 elements. It will always contain the number of elements you specify.

            TArrays on the other hand are dynamic arrays that can resize using Add, Remove etc. You can then also use Num to get the number of elements it contains.

            To create a TArray of ProfileStatsData you would use:

            Code:
            TArray<FProfileStatsData> ProfileInfo;
            The reason you would prepend the "F" to the struct name is because that is a requirement of the Unreal Engine.

            Check out these links for Epic help:

            Unreal Engine - Coding Standards (Tells you about the "F" in struct names)
            https://docs.unrealengine.com/en-US/...ard/index.html

            Unreal Engine - TArrays
            https://docs.unrealengine.com/en-US/...ays/index.html

            Unreal Engine - Structs/Classes:
            https://medium.com/@wojtek.zinczuk/h...s-a14f5e770045

            Comment


              #7
              I am creating a 3 array structs to hold all the players stats, weapon setups and save it to a file for loading when game starts so it just an array that will hold max 3 profiles and 5 loadouts and 75 weapons all ran from 2 fors to get the stats and to write them. Its basically a account manager i making. Until i can get this structure to work rest is on hold. Hey thanks for the input and the tips. Will check those out, thanks.

              Comment


                #8
                From my tests, even an empty lambda causes a leak with Async calls because "this" is always passed by value, seems like the engine makes a copy of the entire object, because the way lambda captures work.

                Apparently there are workaround for this in Cpp17, but I didn't try it to see if changed anything.
                | Savior | USQLite | FSM | Object Pool | Sound Occlusion | Property Transfer | Magic Nodes | MORE |

                Comment


                  #9
                  May "this" be of interest bruno?

                  https://www.nextptr.com/tutorial/ta1...line-of-change

                  It talks about C++11, 14, 17 and 20.

                  You may understand it more than me

                  Comment


                    #10
                    I've already tried those variations of "this" on the lambda... the engine is still leaking memory.
                    Even when not passing in any local vars.

                    Also tried a little experiment:
                    Code:
                    auto* Self = const_cast<UMyObj*>(this);
                    const UPTRINT IntPtr = reinterpret_cast<UPTRINT>(Self);
                    
                    Async( EAsyncExecution::Thread,[&IntPtr] () {/*...*/} );
                    It leaks too!

                    If I do this instead, the code won't leak:
                    Code:
                    FSimpleDelegateGraphTask::CreateAndDispatchWhenReady (
                        FSimpleDelegateGraphTask::FDelegate::CreateStatic( &IMyClass::SomeCall, IntPtr ),
                        GET_STATID(STAT_FMyStats_AsyncTask),
                        nullptr, ENamedThreads::AnyThread
                    );
                    Last edited by BrUnO XaVIeR; 11-06-2020, 10:35 AM.
                    | Savior | USQLite | FSM | Object Pool | Sound Occlusion | Property Transfer | Magic Nodes | MORE |

                    Comment


                      #11
                      There seems to be no unified way in which each async method executes its tasks and allocates its memory.

                      I did have some success recently with FRunnable. I won't post all of the code as it's too long but here is the creation code:

                      Code:
                      class FMyNewRunTask : public FRunnable
                      {
                      protected:
                      FRunnableThread* Thread; //Thread to run the worker on
                      FThreadSafeCounter StopTaskCounter; //Thread safe counter to stop the thread
                      
                      public:
                      FMyNewRunTask(FOnCompletion* InCompleteCallback, bool* bInTaskCompleted)
                      : Thread(nullptr)
                      , StopTaskCounter(0)
                      , CompleteCallback(InCompleteCallback)
                      , bTaskCompleted(bInTaskCompleted)
                      {
                      UE_LOG(LogTemp, Error, TEXT("Task Constructor"));
                      MyNewDataObject = new FMyNewDataObject();
                      }
                      
                      ~FMyNewRunTask()
                      {
                      delete MyNewDataObject;
                      MyNewDataObject = nullptr;
                      UE_LOG(LogTemp, Error, TEXT("Task Destructor"));
                      }
                      then when I want my "Complete Callback" to trigger I use the below code from with the Run() function at the end of processing:

                      Code:
                      AsyncTask(ENamedThreads::GameThread, [this]()
                      {
                      if (this)
                      {
                      if (CompleteCallback->IsBound())
                      {
                      CompleteCallback->Execute(MyNewDataObject);
                      }
                      }
                      });
                      
                      int32 LoopCounter = 0;
                      
                      //Wait for the callback to finish
                      do
                      {
                      //...waiting
                      FPlatformProcess::Sleep(0.1f);
                      LoopCounter++;
                      } while ((*bTaskCompleted) == false || LoopCounter > 100.f /*10 seconds to stop endless loop*/);
                      The delegate is:

                      Code:
                      DECLARE_DELEGATE_OneParam(FOnCompletion, FMyNewDataObject*);
                      I then empty the MyNewDataObject struct on the thread exit().

                      The main variables are:

                      Code:
                      bool* bTaskCompleted = nullptr;
                      FMyNewDataObject* MyNewDataObject;
                      FOnCompletion* CompleteCallback;
                      To run the FRunnable class from the main actor I use:

                      Code:
                      void AMyActor::DoRunnableSomething()
                      {
                      StopTask(); //This function stops the thread if it was run before and deletes it ready for restarting
                      
                      if (!MyNewOnCompletion.IsBound())
                      {
                      MyNewOnCompletion = FOnCompletion::CreateUObject(this, &AMyActor::MyNewCompletionFunction);
                      }
                      
                      bTaskComplete = false;
                      
                      RunTaskWorker = new FMyNewRunTask(&MyNewOnCompletion, &bTaskComplete);
                      RunTaskWorker->Startup();
                      }
                      The complete callback calls this:

                      Code:
                      void AMyActor::MyNewCompletionFunction(FMyNewDataObject* TheData)
                      {
                      TheMainDataObject = *TheData; //This dereferences and copies the data
                      bTaskComplete = true;
                      }
                      The end result at the moment for me is that the data inside the async task produces its data and posts back it pointer to its data. The main actor then copies the data and does what it wants with it. The async object is then free to delete its own version of the data.

                      To delete the data I use this:

                      Code:
                      void AMyActor::Destroyed()
                      {
                      //Empty whatever the struct contains
                      TheMainDataObject.TheIntArray.Empty();
                      TheMainDataObject.TheFloatArray.Empty();
                      TheMainDataObject.TheStructArray.Empty();
                      
                      StopTask(); //Stops the thread worker if it exists and deletes it
                      
                      UE_LOG(LogTemp, Warning, TEXT("Actor got Destroyed!"));
                      
                      Super::Destroyed();
                      }
                      I have hammered away at this for the last month so I am pretty tired of it now. The code right now works so

                      Parts that I am not quite understanding is how I am supposed to delete the async task after the callback has triggerd because if you don't do the "Do...While" then the task deletes before the callback has been called.

                      All opinions are welcome.

                      Thanks.
                      Last edited by 3dev; 11-06-2020, 12:26 PM.

                      Comment


                        #12
                        Code:
                        auto* Self = const_cast<UMyObj*>(this);
                        const UPTRINT IntPtr = reinterpret_cast<UPTRINT>(Self);
                        Async( EAsyncExecution::Thread,[&IntPtr] () {/*...*/} );
                        That actually worked.
                        From inside the thread I get the UObject pointer back:
                        Code:
                        UMyObj* Self = reinterpret_cast<UMyObj*>(IntPtr);
                        What was leaking really hard was actually UE_LOG(...) inside my thread
                        I now have a preprocessor in my logging functions:
                        Code:
                        #if UE_BUILD_SHIPPING
                            return;
                        #endif
                        
                        //...
                        So I let log functions leak in Editor, but it's clean in the actual packaged build.
                        | Savior | USQLite | FSM | Object Pool | Sound Occlusion | Property Transfer | Magic Nodes | MORE |

                        Comment

                        Working...
                        X