
= modular-build =

== Introduction ==

modular-build is a tool for building Native Client's GCC-based
toolchain, written with an eye to being a general purpose build
system.  As a general purpose build system, it has these features:

 * Like Make or Scons, it uses a dependency graph of build targets.

 * Unlike Scons or Make, build targets are directory trees rather than
   individual files.  This means a build target can be a whole source
   tree, such as the gcc source tree, or an install tree, such as the
   result of "make install" on gcc.

 * Like Scons (but unlike Make), it uses file contents, rather than
   timestamps, to determine when a dependency has changed.

Some build targets' directory trees are made by combining trees from
other targets together.  For each, gcc expects to be run from the same
"bin" directory as "ld" (the linker, from binutils), which expects
"bin" to be next to "lib" (containing libc.so/libc.a).  So the final
toolchain directory is created by unioning together the install trees
from these modules: gcc, binutils and libc (newlib or glibc).  These
install trees, and the combined toolchain install tree, are all
separate build targets.


== Comparison with JHBuild ==

modular-build is similar to JHBuild in that it is intended for tying
together multiple autoconf-based modules.

The difference from JHBuild is that modular-build tries harder to do
incremental builds correctly, in a reproducible way, and to track
where build outputs came from.

For example, modular-build saves the install tree from each module
into a separate directory, using "make install DESTDIR=...".  This
makes it easy to find where a file has come from.  For example, to
find which modules build a libgcc, you can do:

$ find out/install | grep libgcc
out/install/full-gcc/lib/gcc/nacl64/4.4.3/32/libgcc.a
out/install/full-gcc/lib/gcc/nacl64/4.4.3/libgcc.a
out/install/full-gcc-glibc/lib/gcc/nacl/4.4.3/libgcc.a
out/install/full-gcc-glibc/lib/gcc/nacl/4.4.3/libgcc_eh.a
out/install/full-gcc-glibc/nacl/lib32/libgcc_s.so
out/install/full-gcc-glibc/nacl/lib32/libgcc_s.so.1
out/install/pre-gcc/lib/gcc/nacl/4.4.3/libgcc.a

In contrast, with JHBuild, the modules' "make install" steps all write
into the same, shared directory.  JHBuild does not track what has been
written into this directory.


== Poor man's sandboxing ==

The ideal way to ensure a reproducible and correctly-incremental build
would be to sandbox individual build commands, so that we can ensure
they do not have undeclared dependencies.  However, there aren't any
sandboxing systems that we can readily use for this.

Instead, we "sandbox" the build of an autoconf module by creating a
directory that contains all of the module's build dependencies (at
least, all of the dependencies that modular-build manages).  We pass
this directory to autoconf's "--prefix" option (plus via some
environment variables).  This directory is called the "input prefix".
A well-behaved module should only look in this directory (plus /usr,
but tracking /usr is out of scope), and should not go trawling the
filesystem, so if the module depends on files that have not been added
into this directory, the undeclared dependency should be caught.

This means that a given install tree tends to be copied multiple times
by modular-build, once per module that depends on it.  For speed, and
to save space, the files are hard-linked rather than copied.


== Basic usage ==

build.py is invoked with a list of build target names.  If no build
targets are specified, it uses all targets.

The "-b" option tells build.py to build targets.  Without this option,
it just prints a list of build targets and whether they would be
rebuilt.

Examples:

$ python build.py
gmp-input: yes
gmp-src: no
gmp: maybe
mpfr-input: yes
mpfr-src: no
mpfr: maybe
binutils-input: yes
binutils-src: no
binutils: maybe
...

"gmp: maybe" indicates that "gmp" will only be rebuilt if its
dependencies change during the course of the rebuild.

$ python build.py -b
...
** building gmp-input
** skipping gmp-src
** skipping gmp
** building mpfr-input
** skipping mpfr-src
** building mpfr
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
...


== Directory layout ==

out/source/X - The source tree for module X.  (Target name: X-src)

out/build/X - Build tree for module X.  This is the directory that
"make" is run in.  This is not actually a build target, since it is
only a temporary directory, not the final output of module X.
Therefore it does not have a target name.

out/install/X - Install tree for module X, as produced by "make
install DESTDIR=...".  (Target name: X)

out/input-prefix/X - Input directory for module X.  This contains the
combined install trees for all of X's dependencies.  When X is
configured, this directory is passed as the "--prefix" argument.
(Target name: X-input)


== Showing a dependency graph using Graphviz ==

To display a graph of build targets and their dependencies, use the
"graph" subcommand.  This outputs a "dot" file which can be processed
with Graphviz's "dot" tool to produce a PDF file:

python build.py graph > graph.dot
dot -Tpdf graph.dot -o graph.pdf
evince graph.pdf


== Shortcuts for quicker rebuilds ==

Suppose we are changing glibc.  A lot of stuff depends on glibc, such
as full-gcc, which takes a long time to build.  We often want to
rebuild (and test) glibc without rebuilding its dependencies.  This is
what the -s (--single) option is for.

In order to test a rebuilt glibc, we usually need to rebuild
glibc_toolchain-input (since that combines glibc with everything else:
binutils, gcc, libsrpc etc.), but we don't need to rebuild full-gcc.
The following command will do that:

$ python build.py glibc glibc_toolchain-input -s -b

If you later re-run this without the -s option, the tool will notice
that glibc's output changed and rebuild full-gcc.

The above command will run the glibc build from scratch
(i.e. "configure" and "make" on a fresh build directory).  In order to
re-use the same build directory, use the --non-pristine (--np) option:

$ python build.py glibc glibc_toolchain-input -s -b --non-pristine

This re-runs "make" but not "configure".

This marks the result of building glibc as non-pristine, so re-running
this without "--non-pristine" will cause glibc to be rebuilt from
scratch.


== Limitations ==

modular-build does not scan source directories in out/source/* when it
is run, so changing source files there does not trigger a rebuild.

This means --non-pristine (--np) tends to be useful only when used
with --single (-s), unless the target has not successfully built
before.
