The primary motivating example is openssl:
Before the change full package build took 1m54s minutes.
After the change full package build takes 59s.
About a 2x speedup.
The difference is visible because openssl builds hundreds of manpages
spawning a perl process per manual in `install` phase. Such a workload
is very easy to parallelize.
Another example would be `autotools`+`libtool` based build system where
install step requires relinking. The more binaries there are to relink
the more gain it will be to do it in parallel.
The change enables parallel installs by default only for buiilds that
already have parallel builds enabled. There is a high chance those build
systems already handle parallelism well but some packages will fail.
Consistently propagated the enableParallelBuilding to:
- cmake (enabled by default, similar to builds)
- ninja (set parallelism explicitly, don't rely on default)
- bmake (enable when requested)
- scons (enable when requested)
- meson (set parallelism explicitly, don't rely on default)
- waf (set parallelism explicitly, don't rely on default)
- qmake-4/5/6 (enable by default, similar to builds)
- xorg (always enable, similar to builds)
Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.
For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.
This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.
A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.
There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.
For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.
Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
ninja build progress output is not line-base, it overwrites the same
line over and over again with its progress reporting. nix is line-based
though, so ninja-based builds have their progress hidden. pipe ninja
output through cat to avoid this.
ninja sources include re2c's output files, so unless we change the sources by applying a patch, re2c is not even launched
anyway, it is not relevant to building docs
Things changed in the Ninja setup-hook:
- Respect installFlags
- Automatically add checkPhase (can be disabled with dontUseNinjaCheck
in the same way as dontUseNinjaBuild and dontUseNinjaInstall). Tests
are only run when "ninja test" exists.
- Error in build phase when build.ninja is missing. We don’t have a
way to fall back to other build methods, so it’s best to be very
clear when we aren’t able to build with ninja
- Set -l flag to 1 when enableParallelBuilding is disabled
This also updates the bootstrap tool builder to LLVM 5, but not the ones
we actually use for bootstrap. I'll make that change in a subsequent commit
so as to provide traceable provenance of the bootstrap tools.
* Add setupHook for meson/ninja build
* libhttpseverywhere: Use meson/ninja setupHooks
* jamomacore: Remove superfluous ninja buildInput
* Remove obsolete ninja buildPhases
These are all handled by ninja's setup hook.
* lean2, xcbuild: fix build with ninja setup hook
Ninja is a runtime dependency here. However, cmake can generate Ninja
build files as well to satisfy the setup hook.
* qtwebengine: fix build with ninja setup hook
Version 1.2.0 is way too old in order to build the latest chromium (29) version,
so let's get it up to date (especially because no other package is referencing
ninja, so it should be non-critical).
The dependency on unzip is not needed here, because GitHub also provides
archives in tar.gz format.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
