Announcement

Collapse
No announcement yet.

New Core Feature: Async Framework (Master Branch and 4.8)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    New Core Feature: Async Framework (Master Branch and 4.8)

    Hi all,

    We started working on a new framework for asynchronous programming, which is located in the Core module. The goal is to simplify the process of writing code that executes asynchronously or in parallel. The Engine already had several mechanisms for this in place for a long time (threads, thread pool, task graph, etc.), but they required a fair amount of boilerplate code.

    The first iteration of this new framework attempts to reduce the amount of boilerplate to an absolute minimum. The two most important additions are the Async<T>() function template and the TFuture<T> type. If you used Java, C# or the newer version of the C++ STL then you may already be familiar with these concepts.


    What is Async<T>?

    The new Async<T> template function allows you to execute functions on another thread without having to write a lot of boilerplate code. There are currently three different execution methods available: TaskGraph, Thread and ThreadPool. All three methods will execute your function asynchronously, allowing the function to run in parallel with the calling thread. Which method you chose depends on the nature of your asynchronous tasks. Functions that are executed through Async<T> will return immediately. This means that the result of the function is actually not available right away, and some mechanism is needed to retrieve the result at a later time. This mechanism is called a Future.


    What is a Future?

    A future is a variable whose value will be set in the future. If you write a function that returns, say, a TFuture<int> instead of just an int, then your function tells its users that the integer return value is not available right away, but it promises to set the value at some time in the future. While your function is computing the result, the caller can meanwhile work on other things, but as soon as the caller attempts to access the actual value of TFuture<int>, its thread will block until that value has actually been set by you. Futures therefore provide a low-level mechanism to return results from functions that are executed asynchronously.


    When should I use Futures?

    Futures are most useful when your code requires results from one or more other functions that execute asynchronously, or in parallel. For example, consider the following scenario where the function Foo() computes and returns a result using three other functions that are computationally expensive:

    Code:
    int Foo()
    {
        int A = CalculateA();
        int B = CalculateB();
        int C = CalculateC();
    
        // other code here
        ....
    
        return A + B + C;
    }
    Note that the computation of the final result requires all of A, B and C. Traditionally, the order of execution would be sequential:

    Code:
    Thread 1:   |____CalculateA____|______CalculateB_____|___________CalculateC___________|__other code__|___A + B + C___|

    With the Async<T>() template function we are able to launch each of CalculateA, CalculateB and CalculateC asynchronously, which allows us to parallelize most of the work:

    Code:
    int Foo()
    {
       Future<int> A = Async<int>(EAsyncExecution::Thread, CalculateA);
       Future<int> B = Async<int>(EAsyncExecution::Thread, CalculateB);
       Future<int> C = Async<int>(EAsyncExecution::Thread, CalculateC);
    
        // other code here
        ....
    
        return A.Get() + B.Get() + C.Get();
    }
    The order of execution now looks something like this:

    Code:
    Thread 1:  |___Async()___|__other code__|///sleep///|___A + B + C___]
    Thread 2:  |____CalculateA____|
    Thread 2:    |_______CalculateB______|
    Thread 3:       |_____________CalculateC____________|

    Note that, here the three functions do not return integer results, but futures that will eventually hold the results. When Foo() has completed all its work and goes on to compute the result, it will block until all of A, B and C are actually available (indicated by "sleep" in the diagram above).


    When should I NOT use Futures?

    When your calling function does not actually care about the results of the asynchronous operations and does not need to block until the operations complete, you should not use futures. You can still execute such units of work asynchronously, and if some other code in your system needs to know about when they complete then it is generally better to use a mechanism using callbacks or delegates or instead. The key here is that the results of the async operations may be needed somewhere, but not in your calling code.


    When should I use TaskGraph vs. Thread vs. ThreadPool for async execution?

    Unreal Engine provides several means of parallelizing execution of tasks.

    The TaskGraph is shared by many other systems in the Engine and is intended for small tasks that are very short-running, never block, and must complete as soon as possible. Launching graph tasks is very cheap as compared to starting up threads, but you must ensure that your code does not block the TaskGraph ever. In particular, you should not set up Async() functions on the TaskGraph that in turn create other Async<T>() calls or may wait on some external event.This is very important, because if all worker threads are waiting then nothing else gets done in the Engine. If your code may block or create other asynchronous calls then use Thread or ThreadPool instead.

    Threads are quite expensive to create and best suited for long running tasks or tasks that may block. Operating systems generally impose limits on the number of threads that can be created, and they also slow down considerably once too many threads are alive at the same time. If you have many tasks (hundreds) or only want to maximize CPU utilization and do not care about all your tasks actually running in parallel at the same time, use ThreadPool instead.

    The ThreadPool is another set of worker threads that is independent from the TaskGraph system. It allows you to queue up an arbitrary number of threads, which will then be completed one after another based on the availability of worker threads. If your tasks do not fit into either TaskGraph or Thread, then execute them here.

    Note: A fourth mechanism for parallel execution, OS processes, is available in the Engine, but it is not exposed in Async<T>(). Use Thread or ThreadPool instead.


    Does this mean my algorithms are parallel now?

    No, futures and Async<T>() are low-level primitives that help reduce the boilerplate code required for asynchronous programming. They do not offer anything for automatically parallelizing your algorithms (although they may be used for a parallel programming library that we might implement in the future, but this is still pie in the sky).


    Does this mean my algorithms are thread-safe now?

    No, you are still responsible for ensuring that any code being executed asynchronously is completely thread-safe. Futures only guarantee thread-safety for the return values of your functions.




    Note: Examples of Async and TFuture can be found in /Runtime/Core/Tests/Async/AsyncTest.cpp. The implementation of Async itself also uses futures.
    Last edited by gmpreussner; 01-13-2015, 07:37 PM.
    Gerke Max Preussner | UE4 Programming Consultant

    Logo
    Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
    Chat with me as gmpreussner on Discord.

    #2
    Some more implementation details for those who care:

    Our implementation separates the read and write side of asynchronous results into two concepts: Futures and Promises. A Future is the object being returned to the caller. It can be used to retrieve a functions return value when it is needed at some time in the future. If the return value is not yet available, the calling thread will block (there is also an option to wait with a timeout). A Promise is used by the called function to write the result into the Future.

    Futures and Promises cannot be copied - they can only be moved. If you move them from one instance to another, the old instance becomes invalid and can no longer be used to set or retrieve result values; this is an optimization. If you wish to share a Future between multiple threads, you can call TFuture.Share() to create a Shared Future. Shared futures are copyable, but do not support the more efficient move semantics.

    The full implementation is in /Runtime/Core/Public/Async/Future.h. Please see the code documentation for further details.

    For a general introduction to the Future/Promise pattern check out this Wikipedia article.
    Gerke Max Preussner | UE4 Programming Consultant

    Logo
    Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
    Chat with me as gmpreussner on Discord.

    Comment


      #3
      As a side note, if you're using this feature be very aware of how C++ lambdas work with variable capture. It's very easy to capture a stack value by reference that you didn't intend to that can lead to race conditions!

      You can capture specific variables rather than all variables (my preference) and choose whether to capture them by-value or by-reference.

      e.g., a simple case:

      Code:
      for (int i = 0; i < SomeStuff.Num(); ++i)
      {
      	auto Future = Async<int>(EAsyncExecution::ThreadPool, [&]
      	{
      		// "i" has been captured by reference! By the time this thing runs on another thread who knows what it will be
      		// We could instead use [=] but that would copy the SomeStuff array which we probably don't want.
      		// We could use [i,&SomeStuff] to capture i by value and SomeStuff by reference. That works as long as we guarantee that the array and its contents won't change while these async operations are in-flight.
      		return DoSlowOperation(SomeStuff[i]);
      	});
      }
      Nick Penwarden, Lead Programmer @ Epic Games

      Comment


        #4
        Thank you Gerke for taking the time to write up this extremely helpful information!

        And thanks Nick P. for the additional info!

        I can't wait to try out the new Future/Async primitives!



        Rama
        100+ UE4 C++ Tutorials on the UE4 Code Wiki, including UE4 Multi-Threading!

        UE4 Marketplace: Melee Weapon Plugin & Compressed Binary Save System Plugin | Rama's C++ AI Jumping Videos | Vertex Snap Editor Plugin

        Visit www.ue4code.com to see lots of videos about my C++ Creations! ♥ Rama

        Comment


          #5
          Very nice! I'm excited to try these, need to set aside some time to seriously dive in. Several algorithms for things I want to do would be nicer in parallel, but I don't have much experience with async in C++ so I've been putting them off.

          Thanks for the warning nick_p!

          Comment


            #6
            Nice Addition, will definitely try it asap!
            [Gamedev programmer at Darewise (Paris) - We are hiring]
            UE4 Git LFS 2.x Source Control Plugin 2.12-beta for UE4.22 - (v1 integrated by default in Beta status since UE4.7)
            UE4 Plastic SCM Source Control Plugin (1.4.5 for UE4.22)

            Comment


              #7
              Very nice! I love the async/await support in C#
              Programmer at Klang Games | Ex-CCP (EVE Valkyrie / Sparc) | @SiggiGG
              Check out my procedural mesh examples

              Comment


                #8
                Very nice addition, giving more freedom on how we can use async primitives will let us do far more stuff!

                Thanks!
                Sr. Engine Programmer @ www.playspace.com - moritzwundke.com
                Remember: be polite and respect other peoples opinions - Join the Unofficial Unreal Discord Channel - Found a bug? Then use the Bug Report Form to get it fixed ^^

                Comment


                  #9
                  Nice addition! Thanks for the info too.

                  Comment


                    #10
                    I wonder, is there a mechanism for async work with non-blocking wait? I.e. an object ticked every frame that checks if its assigned thread/task is finished, and when that's finished, it executes some bound function and uses any data created/calculated by the thread/task to do work on the main thread. I.e. some sort of IO operation during gameplay, where you want to use the results when they're ready, but you don't want to block the game thread with a future (since you have no idea how long this loading will take; the future could halt the main thread for many frames).

                    Comment


                      #11
                      Originally posted by HateDread View Post
                      I wonder, is there a mechanism for async work with non-blocking wait? I.e. an object ticked every frame that checks if its assigned thread/task is finished, and when that's finished, it executes some bound function and uses any data created/calculated by the thread/task to do work on the main thread. I.e. some sort of IO operation during gameplay, where you want to use the results when they're ready, but you don't want to block the game thread with a future (since you have no idea how long this loading will take; the future could halt the main thread for many frames).
                      Unless I misunderstand, you should be able to accomplish what you want with the regular FAsyncTask.

                      Check out the code for sound decompression, it can be used in a similar manner https://github.com/EpicGames/UnrealE...ioDecompress.h
                      Storyteller - An immersive VR audiobook player

                      Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                      Comment


                        #12
                        Originally posted by n00854180t View Post
                        Unless I misunderstand, you should be able to accomplish what you want with the regular FAsyncTask.

                        Check out the code for sound decompression, it can be used in a similar manner https://github.com/EpicGames/UnrealE...ioDecompress.h
                        Thanks for the lead! FAsyncTask looks nice (I'm used to using FRunnables, which have a bit more overhead and setup, so this is cool!)

                        This doesn't handle the 2nd part, however - the execution of work on the main thread once the async portion has been completed. I can of course wrap this task with a tickable object and execute a function pointer once MyTask->IsDone() returns true, but I was hoping for something built-in. Even a function that is called on the main thread once the task has completed (and which I could override in a subclass), such as a virtual void OnTaskCompleted() function. Any ideas?

                        Comment


                          #13
                          Originally posted by HateDread View Post
                          Thanks for the lead! FAsyncTask looks nice (I'm used to using FRunnables, which have a bit more overhead and setup, so this is cool!)

                          This doesn't handle the 2nd part, however - the execution of work on the main thread once the async portion has been completed. I can of course wrap this task with a tickable object and execute a function pointer once MyTask->IsDone() returns true, but I was hoping for something built-in. Even a function that is called on the main thread once the task has completed (and which I could override in a subclass), such as a virtual void OnTaskCompleted() function. Any ideas?
                          Not sure, I don't know enough about the C++ classes to be able to say whether there's something like that. There is a method to check if work is finished on the FAsyncTask, but then you do have to either wrap it in a ticked object or check it in certain intervals.

                          I'd actually like to know if there's a way to set up an event for FAsyncTask myself as that would be a lot cleaner for some things I'm doing as well.
                          Storyteller - An immersive VR audiobook player

                          Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                          Comment


                            #14
                            Originally posted by n00854180t View Post
                            There is a method to check if work is finished on the FAsyncTask, but then you do have to either wrap it in a ticked object or check it in certain intervals. I'd actually like to know if there's a way to set up an event for FAsyncTask myself as that would be a lot cleaner for some things I'm doing as well.
                            We don't have continuations yet, if that's what you mean, but it's on the to-do list.

                            In the meantime, you could pass a delegate as a parameter to the async function, which the async function will execute when it is complete. Make sure that your delegate handler is thread-safe.
                            Gerke Max Preussner | UE4 Programming Consultant

                            Logo
                            Follow me on Github | Goodreads | Linkedin | Pinterest | SlideShare | Twitter.
                            Chat with me as gmpreussner on Discord.

                            Comment


                              #15
                              Originally posted by gmpreussner View Post
                              We don't have continuations yet, if that's what you mean, but it's on the to-do list.

                              In the meantime, you could pass a delegate as a parameter to the async function, which the async function will execute when it is complete. Make sure that your delegate handler is thread-safe.
                              Is there somewhere the engine does this we could look at as an example? I get the concept, just haven't really done much with delegates yet.

                              Thanks for the help
                              Storyteller - An immersive VR audiobook player

                              Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                              Comment

                              Working...
                              X