== Task generators

=== Rule-based task generators (Make-like)

This chapter illustrates the use of rule-based task generators for building simple targets.

==== Declaration and usage

Rule-based task generators are a particular category of task generators producing exactly one task.

The following example shows a task generator producing the file 'foobar.txt' from the project file 'wscript' by executing the command _cp_ to perform a copy:

// rule_simple
[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld( <1>
		rule   = 'cp ${SRC} ${TGT}', <2>
		source = 'wscript', <3>
		target = 'foobar.txt', <4>
	)
---------------

<1> To instantiate a new task generator, remember that all arguments have the form 'key=value'
<2> The attribute _rule_ represents the command to execute in a readable manner (more on this in the next chapters).
<3> Source files, either in a space-delimited string, or in a list of python strings
<4> Target files, either in a space-delimited string, or in a list of python strings

Upon execution, the following output will be observed:

// rules_simple
[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.000s)
'configure' finished successfully (0.021s)
Waf: Entering directory `/tmp/rules_simple/build'
[1/1] foobar.txt: wscript -> build/foobar.txt <1>
10:57:33 runner 'cp ../wscript foobar.txt' <2>
Waf: Leaving directory `/tmp/rules_simple/build'
'build' finished successfully (0.016s)

$ tree
.
|-- build
|   |-- c4che
|   |   |-- build.config.py
|   |   `-- _cache.py
|   |-- config.log
|   `-- foobar.txt
`-- wscript

$ waf <3>
Waf: Entering directory `/tmp/rules_simple/build'
Waf: Leaving directory `/tmp/rules_simple/build'
'build' finished successfully (0.006s)

$ echo " " >> wscript <4>

$ waf
Waf: Entering directory `/tmp/rules_simple/build'
[1/1] foobar.txt: wscript → build/foobar.txt <5>
Waf: Leaving directory `/tmp/rules_simple/build'
'build' finished successfully (0.013s)
---------------

<1> In the first execution, the target is correctly created
<2> Command-lines are only displayed in 'verbose mode' by using the option '-v'
<3> The target is up-to-date, so the task is not executed
<4> Modify the source file in place by appending a space character
<5> Since the source has changed, the target is created once again

The string for the rule also enters in the dependency calculation. If the rule changes, then the task will be recompiled.

==== Rule functions

Rules may be given as expression strings or as python function. The function is assigned to the task class created:

// rule_function
[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	def run(task): <1>
		src = task.inputs[0].abspath() <2>
		tgt = task.outputs[0].abspath() <3>
		cmd = 'cp %s %s' % (src, tgt)
		print(cmd)
		return task.exec_command(cmd) <4>

	bld(
		rule   = run, <5>
		source = 'wscript',
		target = 'same.txt',
	)
---------------

<1> Rule functions take the task instance as parameter.
<2> Sources and targets are represented internally as Node objects bound to the task instance.
<3> Commands are executed from the root of the build directory. Node methods such as 'bldpath' ease the command line creation.
<4> The task class holds a wrapper around subprocess.Popen(...) to execute commands.
<5> Use a function instead of a string expression

The execution trace will be similar to the following:

[source,shishell]
---------------
$ waf distclean configure build
'distclean' finished successfully (0.001s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/rule_function/out'
[1/1] same.txt: wscript -> out/same.txt
cp /tmp/rule_function/wscript /tmp/rule_function/build/same.txt
Waf: Leaving directory `/tmp/rule_function/out'
'build' finished successfully (0.010s)
---------------

The rule function must return a null value (0, None or False) to indicate success, and must generate the files corresponding to the outputs. The rule function is executed by threads internally so it is important to write thread-safe code (cannot search or create node objects).

Unlike string expressions, functions may execute several commands at once.

==== Shell usage

The attribute 'shell' is used to enable the system shell for command execution. A few points are worth keeping in mind when declaring rule-based task generators:

. The Waf tools do not use the shell for executing commands
. The shell is used by default for user commands and custom task generators
. String expressions containing the following symbols `>', `<' or `&' cannot be transformed into functions to execute commands without a shell, even if told to
. In general, it is better to avoid the shell whenever possible to avoid quoting problems (paths having blank characters in the name for example)
. The shell is creating a performance penalty which is more visible on win32 systems.

Here is an example:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(rule='cp ${SRC} ${TGT}', source='wscript', target='f1.txt', shell=False)
	bld(rule='cp ${SRC} ${TGT}', source='wscript', target='f2.txt', shell=True)
---------------

Upon execution, the results will be similar to the following:

[source,shishell]
---------------
$ waf distclean configure build --zones=runner,action
'distclean' finished successfully (0.004s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/rule/out'
23:11:23 action <1>
def f(task):
	env = task.env
	wd = getattr(task, 'cwd', None)
	def to_list(xx):
		if isinstance(xx, str): return [xx]
		return xx
	lst = []
	lst.extend(['cp'])
	lst.extend([a.srcpath(env) for a in task.inputs])
	lst.extend([a.bldpath(env) for a in task.outputs])
	lst = [x for x in lst if x]
	return task.exec_command(lst, cwd=wd)

23:11:23 action
def f(task):
	env = task.env
	wd = getattr(task, 'cwd', None)
	p = env.get_flat
	cmd = ''' cp %s %s ''' % (" ".join([a.srcpath(env) for a in task.inputs]), <2>
		" ".join([a.bldpath(env) for a in task.outputs]))
	return task.exec_command(cmd, cwd=wd)

[1/2] f1.txt: wscript -> out/f1.txt
23:11:23 runner system command -> ['cp', '../wscript', 'f1.txt'] <3>
[2/2] f2.txt: wscript -> out/f2.txt
23:11:23 runner system command ->  cp ../wscript f2.txt
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.017s)
---------------

<1> String expressions are converted to functions (here, without the shell).
<2> Command execution by the shell. Notice the heavy use of string concatenation.
<3> Commands to execute are displayed by calling 'waf --zones=runner'. When called without the shell, the arguments are displayed as a list.

NOTE: For performance and maintainability, try avoiding the shell whenever possible

==== Inputs and outputs

Source and target arguments are optional for make-like task generators, and may point at one or several files at once. Here are a few examples:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld( <1>
		rule   = 'cp ${SRC} ${TGT[0].abspath()} && cp ${SRC} ${TGT[1].abspath()}',
		source = 'wscript',
		target = 'f1.txt f2.txt',
		shell  = True
	)

	bld( <2>
		source = 'wscript',
		rule   = 'echo ${SRC}'
	)

	bld( <3>
		target = 'test.k3',
		rule   = 'echo "test" > ${TGT}',
	)

	bld( <4>
		rule   = 'echo 1337'
	)

	bld( <5>
		rule   = "echo 'task always run'",
		always = True
	)
---------------

<1> Generate 'two files' whenever the input or the rule change. Likewise, a rule-based task generator may have multiple input files.
<2> The command is executed whenever the input or the rule change. There are no declared outputs.
<3> No input, the command is executed whenever it changes
<4> No input and no output, the command is executed only when the string expression changes
<5> No input and no output, the command is executed each time the build is called

For the record, here is the output of the build:

[source,shishell]
---------------
$ waf distclean configure build
'distclean' finished successfully (0.002s)
'configure' finished successfully (0.093s)
Waf: Entering directory `/tmp/rule/out'
[1/5] echo 1337:
1337
[2/5] echo 'task always run':
[3/5] echo ${SRC}: wscript
../wscript
[4/5] f1.txt f2.txt: wscript -> out/f1.txt out/f2.txt
task always run
[5/5] test.k3:  -> out/test.k3
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.049s)

$ waf
Waf: Entering directory `/tmp/rule/out'
[2/5] echo 'task always run':
task always run
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.014s)
---------------

==== Dependencies on file contents

As a second example, we will create a file named 'r1.txt' from the current date. It will be updated each time the build is executed. A second file named 'r2.txt' will be created from 'r1.txt'.

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(
		name   = 'r1', <1>
		target = 'r1.txt',
		rule   = '(date > ${TGT}) && cat ${TGT}', <2>
		always = True, <3>
	)

	bld(
		name   = 'r2', <4>
		target = 'r2.txt',
		rule   = 'cp ${SRC} ${TGT}',
		source = 'r1.txt', <5>
		after  = 'r1', <6>
	)
---------------

<1> Give the task generator a name, it will create a task class of the same name to execute the command
<2> Create 'r1.txt' with the date
<3> There is no source file to depend on and the rule never changes. The task is then set to be executed each time the build is started by using the attribute 'always'
<4> If no name is provided, the rule is used as a name for the task class
<5> Use 'r1.txt' as a source for 'r2.txt'. Since 'r1.txt' was declared before, the dependency will be added automatically ('r2.txt' will be re-created whenever 'r1.txt' changes)
<6> Set the command generating 'r2.txt' to be executed after the command generating 'r1.txt'. The attribute 'after' references task class names, not task generators. Here it will work because rule-based task generator tasks inherit the 'name' attribute

The execution output will be the following:

[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.003s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/rule/out'
[1/2] r1:  -> out/r1.txt
16:44:39 runner system command ->  (date > r1.txt) && cat r1.txt
dom ene 31 16:44:39 CET 2010
[2/2] r2: out/r1.txt -> out/r2.txt
16:44:39 runner system command ->  cp r1.txt r2.txt
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.021s)

$ waf -v
Waf: Entering directory `/tmp/rule/out'
[1/2] r1:  -> out/r1.txt
16:44:41 runner system command ->  (date > r1.txt) && cat r1.txt
dom ene 31 16:44:41 CET 2010
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.016s)
---------------

Although r2 *depends* on 'r1.txt', r2 was not executed in the second build. As a matter of fact, the signature of the task r1 has not changed, and r1 was only set to be executed each time, regardless of its signature. Since the signature of the 'r1.txt' does not change, the signature of r2 will not change either, and 'r2.txt' is considered up-to-date.

We will now illustrate how to make certain that the outputs reflect the file contents and trigger the rebuild for dependent tasks by enabling the attribute 'on_results':

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(
		name   = 'r1',
		target = 'r1.txt',
		rule   = '(date > ${TGT}) && cat ${TGT}',
		always = True,
		on_results = True,
	)

	bld(
		target = 'r2.txt',
		rule   = 'cp ${SRC} ${TGT}',
		source = 'r1.txt',
		after  = 'r1',
	)
---------------

Here 'r2.txt' will be re-created each time:

[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.003s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/rule/out'
[1/2] r1:  -> out/r1.txt
16:59:49 runner system command ->  (date > r1.txt) && cat r1.txt <1>
dom ene 31 16:59:49 CET 2010 <2>
[2/2] r2: out/r1.txt -> out/r2.txt
16:59:49 runner system command ->  cp r1.txt r2.txt
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.020s)

$ waf -v
Waf: Entering directory `/tmp/rule/out'
[1/2] r1:  -> out/r1.txt
16:59:49 runner system command ->  (date > r1.txt) && cat r1.txt
dom ene 31 16:59:49 CET 2010 <3>
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.016s)

$ waf -v
Waf: Entering directory `/tmp/rule/out'
[1/2] r1:  -> out/r1.txt
16:59:53 runner system command ->  (date > r1.txt) && cat r1.txt
dom ene 31 16:59:53 CET 2010 <4>
[2/2] r2: out/r1.txt -> out/r2.txt
16:59:53 runner system command ->  cp r1.txt r2.txt
Waf: Leaving directory `/tmp/rule/out'
'build' finished successfully (0.022s)
---------------

<1> Start with a clean build, both 'r1.txt' and 'r2.txt' are created
<2> Notice the date and time
<3> The second build was executed at the same date and time, so 'r1.txt' has not changed, therefore 'r2.txt' is up to date
<4> The third build is executed at another date and time. Since 'r1.txt' has changed, 'r2.txt' is created once again