Job control is related to the parallelization algorithms used for launching the tasks. While the aim of parallelization is to maximize the amount of tasks in execution, different algorithms may be used in practice
In the NORMAL ordering, task groups are created, and a topological sort is performed in advance. The overall performance penalty for complete builds is usually small, like a few seconds on builds during minutes.
In the JOBCONTROL ordering, groups are created in advance, and a flag indicates the maximum amount of jobs to be used when the consumer threads execute the tasks. This prevents parallelization of tasks which use a lot of resources. For example, linking c++ object files uses a lot of RAM.
In the MAXPARALLEL ordering, Each task holds a list of tasks it must run after (there is only one list of tasks waiting to be executed). Though this version parallelizes tasks very well, it consumes more memory and processing. In practice, Waf may last 20% more on builds when all tasks are up-to-date.
![]() | Warning |
---|---|
Because most task classes use ordering constraints, the maximum parallelization can only be achieved if the constraints between task classes are relaxed, and if all task instances know their predecessors. In the example graph, this was achieved by removing the ordering constraints between the compilation tasks classes and the link tasks classes |
# relax the constraints between cxx and cxx_link (in the build section) import Task Task.TaskBase.classes['cxx'].ext_out = [] # infer the task orderings from input and output nodes import Runner old_refill = Runner.Parallel.refill_task_list def refill_task_list(self): old_refill(self) lst = self.outstanding if lst: for x in lst: for y in lst: for i in x.inputs: for j in y.outputs: if i.id == j.id: x.set_run_after(y) Runner.Parallel.refill_task_list = refill_task_list
From this, we can immediately notice the following:
![]() | Note |
---|---|
In practice, the NORMAL algorithm should be used whenever possible, and the MAXPARALLEL should be used if substantial gains are expected and if the ordering is specified between all tasks. The JOBCONTROL system may be useful for tasks that consume a vast amount of resources. |