Hey there Yang, thanks for confirming the details.
Let us know how it goes. I spoke with one of our devs internally and this was some of their feedback:
- We had to play with the read, decode and write tasks within the DirectoryNode.Extract.cs quite a bit to fine tune this (see lines 250 to 252)
- Looking at the log output they are using the default setup:
- “Using 16 read tasks, 16 decode tasks, 16 write tasks”
- This logic in particular to define the number of write tasks may be of interest since it adapts based on the number of files involved:
- int numWriteTasks = options.NumWriteTasks ?? Math.Min(1 + (int)(directoryNode.Length / (16 * 1024 * 1024)), 16);
- But looking at the logs it’s still falling back to 16 currently which is the default.
- We would advise to consider tweaking with these three values, maybe scaling more aggressively depending on the number of files involved, although we did hit OOM issues so they will need to be careful how far they push things.
Julian