How does the engine do work? (Under the hood; the worker threads, task balancing)

Hey guys,

I’m wondering about some of the action under the hood that would take quite a while to read through and understand (happy to do it, but just thought I’d ask here first / as well).

For example;

  • What does each thread that the engine spawns actually do? Worker threads, pool threads. (If you ignore the OpenCL Worker, the OpenCL threads, and the Object Sorter threads, this is still a lot. What do they all do?
  • I’ve seen a lot of talk about task schedulers and task-based work being scalable to thread counts, since a dynamic system and/or library (i.e. in OpenMP) can allocate the ‘tasks’ to threads as needed, no matter how many there are. Is this how the UE4 TaskGraph system works, or is that separate? Is there a task-based system in place? What can we do with it or modify in it?
  • Further to the above; how is it suggested that significant workloads be organized? If it were a small engine done in-house I could imagine deciding to split everything into tasks where possible and writing a scheduler/load balancer, but it’s a bit harder when you’re staring a huge engine in the face. Alternatively one could create a thread for each significant subsystem, but we’re already talking way more threads than the CPU has cores, and it can’t scale in the future.

Any insight would be awesome.