== Advanced scenarios

This chapter demonstrates a few examples of the waf library for more complicated and less common scenarios.

=== Project organization

==== Building the compiler first[[build_compiler_first]]

The example below demonstrates how to build a compiler which is used for building the remaining targets. The requirements are the following:

. Create the compiler and all its intermediate tasks
. Re-use the compiler in a second build step
. The compiler will transform '.src' files into '.cpp' files, which will be processed too
. Call the compiler again if it was rebuilt (add the dependency on the compiler)

The first thing to do is to write the expected user script:

// scenarios_compiler
[source,python]
---------------
top = '.'
out = 'build'

def configure(ctx):
    ctx.load('g++')
    ctx.load('src2cpp', tooldir='.')

def build(ctx):
    ctx.program( <1>
        source = 'comp.cpp',
        target = 'comp')

    ctx.add_group() <2>

    ctx.program(
        source = 'main.cpp a.src', <3>
        target = 'foo')
---------------

<1> Build the compiler first, it will result in a binary named _comp_
<2> Add a new build group to make certain the compiler is complete before processing the next tasks
<3> The file 'a.src' is to be transformed by 'comp' into 'a.cpp'

The code for the _src → cpp_ conversion will be the following:

[source,python]
---------------
from waflib.Task import Task
class src2cpp(Task): <1>
    run_str = '${SRC[0].abspath()} ${SRC[1].abspath()} ${TGT}'
    color   = 'PINK'

from waflib.TaskGen import extension

@extension('.src')
def process_src(self, node): <2>
    tg = self.bld.get_tgen_by_name('comp') <3>
    comp = tg.link_task.outputs[0]
    tsk = self.create_task('src2cpp', [comp, node], node.change_ext('.cpp')) <4>
    self.source.extend(tsk.outputs) <5>
---------------

<1> Declare a new task class for processing the source file by our compiler
<2> Files of extension '.src' are to be processed by this method
<3> Obtain a reference on the task generator producing the compiler
<4> Create the task 'src → cpp', the compiler being as used as the first source file
<5> Add the generated 'cpp' file to be processed too

The compilation results will be the following:

[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.006s)
Setting top to                           : /tmp/scenarios_compiler
Setting out to                           : /tmp/scenarios_compiler/build
Checking for program g++,c++             : /usr/bin/g++
Checking for program ar                  : /usr/bin/ar
'configure' finished successfully (0.118s)
Waf: Entering directory `/tmp/scenarios_compiler/build'
[1/5] cxx: comp.cpp -> build/comp.cpp.0.o
10:21:14 runner ['/usr/bin/g++', '../comp.cpp', '-c', '-o', 'comp.cpp.0.o']
[2/5] cxxprogram: build/comp.cpp.0.o -> build/comp <1>
10:21:14 runner ['/usr/bin/g++', 'comp.cpp.0.o', '-o', '/tmp/scenarios_compiler/build/comp']
[3/5] cxx: main.cpp -> build/main.cpp.1.o
10:21:15 runner ['/usr/bin/g++', '../main.cpp', '-c', '-o', 'main.cpp.1.o']
[4/5] src2cpp: a.src -> build/a.cpp
10:21:15 runner ['/tmp/scenarios_compiler/build/comp', '../a.src', 'a.cpp'] <2>
[5/5] cxxprogram: build/main.cpp.1.o -> build/foo <3>
10:21:15 runner ['/usr/bin/g++', 'main.cpp.1.o', '-o', 'foo']
Waf: Leaving directory `/tmp/scenarios_compiler/build'
'build' finished successfully (1.009s)
---------------

<1> Creation of the 'comp' program
<2> Use the compiler to generate 'a.cpp'
<3> Compile and link 'a.cpp' and 'main.cpp' into the program 'foo'

NOTE: When `waf --targets=foo' is called, the task generator `comp' will create its tasks too (task generators from previous groups are processed).

==== Providing arbitrary configuration files

A file is copied into the build directory before the build starts. The build may use this file for building other targets.

// scenarios_impfile
[source,python]
---------------
cfg_file = 'somedir/foo.txt'

def configure(conf):

    orig = conf.root.find_node('/etc/fstab')
    txt = orig.read() <1>

    dest = conf.bldnode.make_node(cfg_file)
    dest.parent.mkdir() <2>
    dest.write(txt) <3>

    conf.env.append_value('cfg_files', dest.abspath()) <4>

def build(ctx):
    ctx(rule='cp ${SRC} ${TGT}', source=cfg_file, target='bar.txt')
---------------

<1> Read the file '/etc/fstab'
<2> Create the destination directory in case it does not already exist
<3> Create a new file in the build directory
<4> Mark the output as a configuration file so it can be used during the build

The execution output will be the following:

[source,shishell]
---------------
$ waf configure build
Setting top to     : /tmp/scenarios_impfile
Setting out to     : /tmp/scenarios_impfile/build
'configure' finished successfully (0.003s)
Waf: Entering directory `/tmp/scenarios_impfile/build'
[1/1] bar.txt: build/somedir/foo.txt -> build/bar.txt
Waf: Leaving directory `/tmp/scenarios_impfile/build'
'build' finished successfully (0.008s)

$ tree
.
|-- build
|   |-- bar.txt
|   |-- c4che
|   |   |-- build.config.py
|   |   `-- _cache.py
|   |-- config.log
|   `-- somedir
|       `-- foo.txt
`-- wscript
---------------



=== Mixing extensions and C/C++ features

==== Files processed by a single task generator

Now let's illustrate the @extension decorator on idl file processing. Files with .idl extension are processed to produce .c and .h files (`foo.idl` → `foo.c` + `foo.h`). The .c files must be compiled after being generated.

First, here is the declaration expected in user scripts:

// scenarios_idl
[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
    conf.load('g++')

def build(bld):
    bld.program(
        source = 'foo.idl main.cpp',
        target = 'myapp'
        )
---------------

The file +foo.idl+ is listed as a source. It will be processed to +foo.cpp+ and compiled and linked with +main.cpp+

Here is the code to support this scenario:

[source,python]
---------------
from waflib.Task import Task
from waflib.TaskGen import extension

class idl(Task):
    run_str = 'cp ${SRC} ${TGT[0].abspath()} && touch ${TGT[1].abspath()}' <1>
    color   = 'BLUE'
    ext_out = ['.h'] <2>

@extension('.idl')
def process_idl(self, node):
    cpp_node = node.change_ext('.cpp')
    hpp_node = node.change_ext('.hpp')
    self.create_task('idl', node, [cpp_node, hpp_node]) <3>
    self.source.append(cpp_node) <4>
---------------

<1> Dummy command for demonstration purposes. In practice the rule to use would be like _omniidl -bcxx $\{SRC} -C$\{TGT}_
<2> Because the idl task produces headers, it must be executed before any other +cpp+ file is compiled
<3> Create the task from the '.idl' extension.
<4> Reinject the file to compile by the C++ compiler

The execution results will be the following:

[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.002s)
Setting top to   : /tmp/scenarios_idl
Setting out to   : /tmp/scenarios_idl/build
Checking for program g++,c++   : /usr/bin/g++
Checking for program ar        : /usr/bin/ar
'configure' finished successfully (0.072s)
Waf: Entering directory `/tmp/scenarios_idl/build'
[1/4] idl: foo.idl -> build/foo.cpp build/foo.hpp
19:47:11 runner 'cp ../foo.idl foo.cpp && touch foo.hpp'
[2/4] cxx: main.cpp -> build/main.cpp.0.o
19:47:11 runner ['/usr/bin/g++', '-I.', '-I..', '../main.cpp', '-c', '-o', 'main.cpp.0.o']
[3/4] cxx: build/foo.cpp -> build/foo.cpp.0.o
19:47:11 runner ['/usr/bin/g++', '-I.', '-I..', 'foo.cpp', '-c', '-o', 'foo.cpp.0.o']
[4/4] cxxprogram: build/main.cpp.0.o build/foo.cpp.0.o -> build/myapp
19:47:11 runner ['/usr/bin/g++', 'main.cpp.0.o', 'foo.cpp.0.o', '-o', 'myapp']
Waf: Leaving directory `/tmp/scenarios_idl/build'
'build' finished successfully (0.149s)
---------------

NOTE: The drawback of this declaration is that the source files produced by the idl transformation can be used by only one task generator.

==== Resources shared by several task generators

Let's suppose now that the idl outputs will be shared by several task generators. We will first start by writing the expected user script:

// scenarios_idl2
[source,python]
---------------
top = '.'
out = 'out'

def configure(ctx):
    ctx.load('g++')

def build(ctx):
    ctx( <1>
        source   = 'notify.idl',
        name     = 'idl_gen')

    ctx.program( <2>
        source   = ['main.cpp'],
        target   = 'testprog',
        includes = '.',
        add_idl  = 'idl_gen') <3>
---------------

<1> Process an idl file in a first task generator. Name this task generator 'idl_gen'
<2> Somewhere else (maybe in another script), another task generator will use the source generated by the idl processing
<3> Reference the idl processing task generator by the name 'idl_gen'.

The code to support this scenario will be the following:

[source,python]
---------------
from waflib.Task import Task
from waflib.TaskGen import feature, before_method, extension

class idl(Task):
	run_str = 'cp ${SRC} ${TGT[0].abspath()} && touch ${TGT[1].abspath()}'
	color   = 'BLUE'
	ext_out = ['.h'] <1>

@extension('.idl')
def process_idl(self, node):
	cpp_node = node.change_ext('.cpp')
	hpp_node = node.change_ext('.hpp')
	self.create_task('idl', node, [cpp_node, hpp_node])
	self.more_source = [cpp_node] <2>

@feature('*')
@before_method('process_source') <3>
def process_add_source(self):
	for x in self.to_list(getattr(self, 'add_idl', [])): <4>
		y = self.bld.get_tgen_by_name(x)
		y.post() <5>
		if getattr(y, 'more_source', None):
			self.source.extend(y.more_source) <6>
---------------

<1> The idl processing must be performed before any C++ task is executed
<2> Bind the output file to a new attribute
<3> Add the source from another task generator object
<4> Process _add_idl_, finding the other task generator
<5> Ensure that the other task generator has created its tasks
<6> Update the source list

The task execution output will be very similar to the output from the first example:

[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.007s)
Setting top to  : /tmp/scenarios_idl2
Setting out to  : /tmp/scenarios_idl2/build
Checking for program g++,c++    : /usr/bin/g++
Checking for program ar         : /usr/bin/ar
'configure' finished successfully (0.080s)
Waf: Entering directory `/tmp/scenarios_idl2/build'
[1/4] idl: foo.idl -> build/foo.cpp build/foo.hpp
20:20:24 runner 'cp ../foo.idl foo.cpp && touch foo.hpp'
[2/4] cxx: main.cpp -> build/main.cpp.1.o
20:20:24 runner ['/usr/bin/g++', '-I.', '-I..', '../main.cpp', '-c', '-o', 'main.cpp.1.o']
[3/4] cxx: build/foo.cpp -> build/foo.cpp.1.o
20:20:24 runner ['/usr/bin/g++', '-I.', '-I..', 'foo.cpp', '-c', '-o', 'foo.cpp.1.o']
[4/4] cxxprogram: build/main.cpp.1.o build/foo.cpp.1.o -> build/testprog
20:20:24 runner ['/usr/bin/g++', 'main.cpp.1.o', 'foo.cpp.1.o', '-o', 'testprog']
Waf: Leaving directory `/tmp/scenarios_idl2/build'
'build' finished successfully (0.130s)
---------------

=== Task generator methods

==== Replacing particular attributes

In general, task generator attributes are not replaced, so the following is not going to be compile +main.c+:

[source,python]
---------------
bld.env.FOO = '/usr/includes'
bld.env.MAIN = 'main.c'
bld(
    features = 'c cprogram',
    source   = '${MAIN}',
    target   = 'app',
    includes = '. ${FOO}')
---------------

This design decision is motivated by two main reasons:

. Processing the attributes has a negative performance impact
. For consistency all attributes would have to be processed

Nevertheless, it is we will demonstrate how to provide Waf with a method to process some attributes. To add a new task generator method, it is necessary to think about its integration with other methods: is there a particular order? The answer is yes, for example, the source attribute is used to create the compilation tasks. To display what methods are in use, execute Waf with the following logging key:

[source,shishell]
---------------
$ waf --zones=task_gen
...
19:20:51 task_gen posting task_gen 'app' declared in 'scenarios_expansion' <1>
19:20:51 task_gen -> process_rule (9232720) <2>
19:20:51 task_gen -> process_source (9232720)
19:20:51 task_gen -> apply_link (9232720)
19:20:51 task_gen -> apply_objdeps (9232720)
19:20:51 task_gen -> process_use (9232720)
19:20:51 task_gen -> propagate_uselib_vars (9232720)
19:20:51 task_gen -> apply_incpaths (9232720)
19:20:51 task_gen posted app
---------------

<1> Task generator execution
<2> Method name and task generator id in parentheses

From the method list, we find that *process_rule* and *process_source* are processing the _source_ attribute. The _includes_ attribute is processed by *apply_incpaths*.

// scenarios_expansion
[source,python]
---------------
from waflib import Utils, TaskGen
@TaskGen.feature('*') <1>
@TaskGen.before('process_source', 'process_rule', 'apply_incpaths') <2>
def transform_strings(self):
	for x in 'includes source'.split(): <3>
		val = getattr(self, x, None)
		if val:
			if isinstance(val, str):
				setattr(self, x, Utils.subst_vars(val, self.env)) <4>
			elif isinstance(val, list):
				for i in xrange(len(val)):
					if isinstance(val[i], str):
						val[i] = Utils.subst_vars(val[i], self.env)
---------------

<1> Execute this method in all task generators
<2> Methods to take into account
<3> Iterate over all interesting attributes
<4> Substitute the attributes

==== Inserting special include flags

A scenario that appears from times to times in C/C++ projects is the need to insert specific flags before others, regardless of how flags are usually processed. We will now consider the following case: execute all C++ compilations with the flag `-I.' in first position (before any other include).

First, a look at the definition of the C++ compilation rule shows that the variable 'INCPATHS' contains the include flags:

[source,python]
---------------
class cxx(Task.Task):
    color   = 'GREEN'
    run_str = '${CXX} ${CXXFLAGS} ${CPPPATH_ST:INCPATHS} ${CXX_SRC_F}${SRC} ${CXX_TGT_F}${TGT}'
    vars    = ['CXXDEPS']
    ext_in  = ['.h']
    scan    = c_preproc.scan
---------------

Those include flags are set by the method 'apply_incpaths'. The trick is then to modify 'INCPATHS' after that method has been executed:

// scenarios_incflags
[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
    conf.load('g++')

def build(bld):
    bld.program(features='cxx cxxprogram', source='main.cpp', target='test')

from waflib.TaskGen import after, feature

@feature('cxx')
@after_method('apply_incpaths')
def insert_blddir(self):
    self.env.prepend_value('INCPATHS', '.')
---------------

A related case is how to add the top-level directory containing a configuration header:

[source,python]
---------------
@feature('cxx')
@after_method('apply_incpaths', 'insert_blddir')
def insert_srcdir(self):
    path = self.bld.srcnode.abspath()
    self.env.prepend_value('INCPATHS', path)
---------------


=== Custom tasks

==== Force the compilation of a particular task

In some applications, it may be interesting to keep track of the date and time of the last build. In C this may be done by using the macros `__DATE__' and `__TIME__', for example, the following +about.c+ file will contain:

[source,c]
---------------
void ping() {
    printf("Project compiled: %s %s\n", __DATE__, __TIME__);
}
---------------

The files are only compiled when they change though, so it is necessary to find a way to force the +about.c+ recompilation. To sum up, the compilation should be performed whenever:

. One of the c files of the project is compiled
. The link flags for any task change
. The link task including the object for our macro is removed

To illustrate this behaviour, we will now set up a project will use various c files:

// scenarios_end
[source,python]
---------------
def options(opt):
    opt.load('compiler_c')

def configure(conf):
    conf.load('compiler_c')

def build(bld):
    bld.program(
        source   = 'main.c about.c',
        target   = 'app',
        includes = '.',
        use      = 'my_static_lib')

    bld.stlib(
        source   = 'test_staticlib.c',
        target   = 'my_static_lib')
---------------

The main file will just call the function _ping_ defined +about.c+ to display the date and time:

[source,c]
---------------
#include "a.h"

int main() {
    ping();
    return 0;
}
---------------

The task method _runnable_status_ must be overridden to take into account the dependencies:

[source,python]
---------------
import os
from waflib import Task
def runnable_status(self):
    if self.inputs[0].name == 'about.c': <1>
        h = 0 <2>
        for g in self.generator.bld.groups:
            for tg in g:
                if isinstance(tg, TaskBase):
                    continue <3>

                h = hash((self.generator.bld.hash_env_vars(self.generator.env, ['LINKFLAGS']), h))
                for tsk in getattr(tg, 'compiled_tasks', []): # all .c or .cpp compilations
                    if id(tsk) == id(self):
                        continue
                    if not tsk.hasrun:
                        return Task.ASK_LATER
                    h = hash((tsk.signature(), h)) <4>
        self.env.CCDEPS = h

        try:
            os.stat(self.generator.link_task.outputs[0].abspath()) <5>
        except:
            return Task.RUN_ME

    return Task.Task.runnable_status(self) <6>

from waflib.Tools.c import c <7>
c.runnable_status = runnable_status
---------------

<1> If the task processes +about.c+
<2> Define a hash value that the task will depend on (CCDEPS)
<3> Iterate over all task generators of the project
<4> Hash the link flags and the signatures of all other compilation tasks
<5> Make sure to execute the task if it was never executed before
<6> Normal behaviour
<7> Modify the 'c' task class

The execution will produce the following output:

[source,shishell]
---------------
$ waf
Waf: Entering directory `/tmp/scenarios_end/build'
[2/5] c: test_staticlib.c -> build/test_staticlib.c.1.o
[3/5] cstlib: build/test_staticlib.c.1.o -> build/libmy_static_lib.a
[4/5] c: about.c -> build/about.c.0.o
[5/5] cprogram: build/main.c.0.o build/about.c.0.o -> build/app
Waf: Leaving directory `/tmp/scenarios_end/build' <1>
'build' finished successfully (0.088s)

$ ./build/app
Project compiled: Jul 25 2010 14:05:30

$ echo " " >> main.c <2>

$ waf
Waf: Entering directory `/tmp/scenarios_end/build'
[1/5] c: main.c -> build/main.c.0.o
[4/5] c: about.c -> build/about.c.0.o <3>
[5/5] cprogram: build/main.c.0.o build/about.c.0.o -> build/app
Waf: Leaving directory `/tmp/scenarios_end/build'
'build' finished successfully (0.101s)

$ ./build/app
Project compiled: Jul 25 2010 14:05:49
---------------

<1> All files are compiled on the first build
<2> The file +main.c+ is modified
<3> The build generates +about.c+ again to update the build time string

==== A compiler producing source files with names unknown in advance

The requirements for this problem are the following:

. A compiler *creates source files* (one .src file -> several .c files)
. The source file names to create are *known only when the compiler is executed*
. The compiler is slow so it should run *only when absolutely necessary*
. Other tasks will *depend on the generated files* (compile and link the .c files into a program)

To do this, the information on the source files must be shared between the build executions.

// scenarios_unknown
[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
    conf.load('gcc')
    conf.load('mytool', tooldir='.')

def build(bld):
    bld.env.COMP = bld.path.find_resource('evil_comp.py').abspath() <1>
    bld.stlib(source='x.c foo.src', target='astaticlib') <2>
---------------

<1> Compiler path
<2> An example, having a _.src_ file

The contents of _mytool_ will be the following:

[source,python]
---------------
import os
from waflib import Task, Utils, Context
from waflib.Utils import subprocess
from waflib.TaskGen import extension

@extension('.src')
def process_shpip(self, node): <1>
    self.create_task('src2c', node)

class src2c(Task.Task):
    color = 'PINK'
    quiet = True <2>
    ext_out = ['.h'] <3>

    def run(self):
        cmd = '%s %s' % (self.env.COMP, self.inputs[0].abspath())
        n = self.inputs[0].parent.get_bld()
        n.mkdir()
        cwd = n.abspath()
        out = self.generator.bld.cmd_and_log(cmd, cwd=cwd, quiet=Context.STDOUT) <4>

        out = Utils.to_list(out)
        self.outputs = [self.generator.path.find_or_declare(x) for x in out]
        self.generator.bld.raw_deps[self.uid()] = [self.signature()] + self.outputs <5>
        self.add_c_tasks(self.outputs) <6>

    def add_c_tasks(self, lst):
        self.more_tasks = []
        for node in lst:
            if node.name.endswith('.h'):
                continue
            tsk = self.generator.create_compiled_task('c', node)
            self.more_tasks.append(tsk) <7>

            tsk.env.append_value('INCPATHS', [node.parent.abspath()])

            if getattr(self.generator, 'link_task', None): <8>
                self.generator.link_task.set_run_after(tsk)
                self.generator.link_task.inputs.append(tsk.outputs[0])

    def runnable_status(self):
        ret = super(src2c, self).runnable_status()
        if ret == Task.SKIP_ME:

            lst = self.generator.bld.raw_deps[self.uid()]
            if lst[0] != self.signature():
                return Task.RUN_ME

            nodes = lst[1:]
            for x in nodes:
                try:
                    os.stat(x.abspath())
                except:
                    return Task.RUN_ME

            nodes = lst[1:]
            self.set_outputs(nodes)
            self.add_c_tasks(nodes) <9>

        return ret
---------------

<1> The processing will be delegated to the task
<2> Disable the warnings raised when a task has no outputs
<3> Make certain the processing will be executed before any task using _.h_ files
<4> When the task is executed, collect the process stdout which contains the generated file names
<5> Store the output file nodes in a persistent cache
<6> Create the tasks to compile the outputs
<7> The c tasks will be processed after the current task is done. This does not mean that the c tasks will always be executed.
<8> If the task generator of the _src_ file has a link task, set the build order
<9> When this task can be skipped, force the dynamic c task creation

The output will be the following:

[source,shishell]
---------------
$ waf distclean configure build build
'distclean' finished successfully (0.006s)
Setting top to  : /tmp/scenarios_unknown
Setting out to  : /tmp/scenarios_unknown/build
Checking for program gcc,cc              : /usr/bin/gcc
Checking for program ar                  : /usr/bin/ar
'configure' finished successfully (0.115s)
Waf: Entering directory `/tmp/scenarios_unknown/build'
[1/3] src2c: foo.src
[2/5] c: build/shpip/a12.c -> build/shpip/a12.c.0.o
[3/5] c: build/shpop/a13.c -> build/shpop/a13.c.0.o
[4/5] c: x.c -> build/x.c.0.o
[5/5] cstlib: build/x.c.0.o build/shpip/a12.c.0.o build/shpop/a13.c.0.o -> build/libastaticlib.a
Waf: Leaving directory `/tmp/scenarios_unknown/build'
'build' finished successfully (0.188s)
Waf: Entering directory `/tmp/scenarios_unknown/build'
Waf: Leaving directory `/tmp/scenarios_unknown/build'
'build' finished successfully (0.013s)
---------------