2017-01-22 20:52:35 +00:00
|
|
|
<chapter xmlns="http://docbook.org/ns/docbook"
|
|
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
|
|
xml:id="chap-cross">
|
2018-05-01 23:54:21 +00:00
|
|
|
<title>Cross-compilation</title>
|
|
|
|
<section xml:id="sec-cross-intro">
|
2017-01-22 20:52:35 +00:00
|
|
|
<title>Introduction</title>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
2017-01-22 20:52:35 +00:00
|
|
|
<para>
|
2019-03-09 05:07:11 +00:00
|
|
|
"Cross-compilation" means compiling a program on one machine for another
|
|
|
|
type of machine. For example, a typical use of cross-compilation is to
|
|
|
|
compile programs for embedded devices. These devices often don't have the
|
|
|
|
computing power and memory to compile their own programs. One might think
|
|
|
|
that cross-compilation is a fairly niche concern. However, there are
|
|
|
|
significant advantages to rigorously distinguishing between build-time and
|
2019-03-20 22:21:00 +00:00
|
|
|
run-time environments! Significant, because the benefits apply even when one
|
|
|
|
is developing and deploying on the same machine. Nixpkgs is increasingly
|
|
|
|
adopting the opinion that packages should be written with cross-compilation
|
|
|
|
in mind, and nixpkgs should evaluate in a similar way (by minimizing
|
|
|
|
cross-compilation-specific special cases) whether or not one is
|
|
|
|
cross-compiling.
|
2017-01-22 20:52:35 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2018-05-01 23:54:21 +00:00
|
|
|
This chapter will be organized in three parts. First, it will describe the
|
|
|
|
basics of how to package software in a way that supports cross-compilation.
|
|
|
|
Second, it will describe how to use Nixpkgs when cross-compiling. Third, it
|
|
|
|
will describe the internal infrastructure supporting cross-compilation.
|
2017-01-22 20:52:35 +00:00
|
|
|
</para>
|
2018-05-01 23:54:21 +00:00
|
|
|
</section>
|
2017-01-22 20:52:35 +00:00
|
|
|
<!--============================================================-->
|
2018-05-01 23:54:21 +00:00
|
|
|
<section xml:id="sec-cross-packaging">
|
2017-02-08 21:43:52 +00:00
|
|
|
<title>Packaging in a cross-friendly manner</title>
|
2017-01-22 20:52:35 +00:00
|
|
|
|
2019-03-20 22:21:00 +00:00
|
|
|
<section xml:id="ssec-cross-platform-parameters">
|
2018-05-01 23:54:21 +00:00
|
|
|
<title>Platform parameters</title>
|
2017-09-15 12:16:22 +00:00
|
|
|
|
2018-05-01 23:54:21 +00:00
|
|
|
<para>
|
2019-03-09 05:07:11 +00:00
|
|
|
Nixpkgs follows the
|
|
|
|
<link
|
2018-11-19 05:23:22 +00:00
|
|
|
xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Configure-Terms.html">conventions
|
2019-03-09 05:07:11 +00:00
|
|
|
of GNU autoconf</link>. We distinguish between 3 types of platforms when
|
|
|
|
building a derivation: <wordasword>build</wordasword>,
|
|
|
|
<wordasword>host</wordasword>, and <wordasword>target</wordasword>. In
|
|
|
|
summary, <wordasword>build</wordasword> is the platform on which a package
|
|
|
|
is being built, <wordasword>host</wordasword> is the platform on which it
|
|
|
|
will run. The third attribute, <wordasword>target</wordasword>, is relevant
|
|
|
|
only for certain specific compilers and build tools.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
2017-09-15 12:16:22 +00:00
|
|
|
|
2018-05-01 23:54:21 +00:00
|
|
|
<para>
|
|
|
|
In Nixpkgs, these three platforms are defined as attribute sets under the
|
2018-10-02 19:59:59 +00:00
|
|
|
names <literal>buildPlatform</literal>, <literal>hostPlatform</literal>,
|
|
|
|
and <literal>targetPlatform</literal>. They are always defined as
|
|
|
|
attributes in the standard environment. That means one can access them
|
|
|
|
like:
|
2018-05-01 23:54:21 +00:00
|
|
|
<programlisting>{ stdenv, fooDep, barDep, .. }: ...stdenv.buildPlatform...</programlisting>
|
|
|
|
.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>buildPlatform</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
The "build platform" is the platform on which a package is built. Once
|
|
|
|
someone has a built package, or pre-built binary package, the build
|
2018-11-19 05:23:22 +00:00
|
|
|
platform should not matter and can be ignored.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>hostPlatform</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
The "host platform" is the platform on which a package will be run. This
|
|
|
|
is the simplest platform to understand, but also the one with the worst
|
|
|
|
name.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>targetPlatform</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
The "target platform" attribute is, unlike the other two attributes, not
|
|
|
|
actually fundamental to the process of building software. Instead, it is
|
|
|
|
only relevant for compatibility with building certain specific compilers
|
|
|
|
and build tools. It can be safely ignored for all other packages.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
The build process of certain compilers is written in such a way that the
|
|
|
|
compiler resulting from a single build can itself only produce binaries
|
2018-11-19 05:23:22 +00:00
|
|
|
for a single platform. The task of specifying this single "target
|
2019-03-09 05:07:11 +00:00
|
|
|
platform" is thus pushed to build time of the compiler. The root cause
|
|
|
|
of this is that the compiler (which will be run on the host) and the
|
|
|
|
standard library/runtime (which will be run on the target) are built by
|
|
|
|
a single build process.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
There is no fundamental need to think about a single target ahead of
|
|
|
|
time like this. If the tool supports modular or pluggable backends, both
|
|
|
|
the need to specify the target at build time and the constraint of
|
|
|
|
having only a single target disappear. An example of such a tool is
|
|
|
|
LLVM.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Although the existence of a "target platfom" is arguably a historical
|
|
|
|
mistake, it is a common one: examples of tools that suffer from it are
|
|
|
|
GCC, Binutils, GHC and Autoconf. Nixpkgs tries to avoid sharing in the
|
|
|
|
mistake where possible. Still, because the concept of a target platform
|
|
|
|
is so ingrained, it is best to support it as is.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The exact schema these fields follow is a bit ill-defined due to a long and
|
|
|
|
convoluted evolution, but this is slowly being cleaned up. You can see
|
|
|
|
examples of ones used in practice in
|
|
|
|
<literal>lib.systems.examples</literal>; note how they are not all very
|
|
|
|
consistent. For now, here are few fields can count on them containing:
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>system</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
This is a two-component shorthand for the platform. Examples of this
|
|
|
|
would be "x86_64-darwin" and "i686-linux"; see
|
2018-11-19 05:23:22 +00:00
|
|
|
<literal>lib.systems.doubles</literal> for more. The first component
|
2019-03-09 05:07:11 +00:00
|
|
|
corresponds to the CPU architecture of the platform and the second to
|
|
|
|
the operating system of the platform (<literal>[cpu]-[os]</literal>).
|
|
|
|
This format has built-in support in Nix, such as the
|
2018-05-01 23:54:21 +00:00
|
|
|
<varname>builtins.currentSystem</varname> impure string.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>config</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
2019-03-09 05:07:11 +00:00
|
|
|
This is a 3- or 4- component shorthand for the platform. Examples of
|
|
|
|
this would be <literal>x86_64-unknown-linux-gnu</literal> and
|
2018-11-19 05:23:22 +00:00
|
|
|
<literal>aarch64-apple-darwin14</literal>. This is a standard format
|
|
|
|
called the "LLVM target triple", as they are pioneered by LLVM. In the
|
|
|
|
4-part form, this corresponds to
|
|
|
|
<literal>[cpu]-[vendor]-[os]-[abi]</literal>. This format is strictly
|
2019-03-09 05:07:11 +00:00
|
|
|
more informative than the "Nix host double", as the previous format
|
|
|
|
could analogously be termed. This needs a better name than
|
2018-05-01 23:54:21 +00:00
|
|
|
<varname>config</varname>!
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>parsed</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
2019-03-09 05:07:11 +00:00
|
|
|
This is a Nix representation of a parsed LLVM target triple with
|
|
|
|
white-listed components. This can be specified directly, or actually
|
|
|
|
parsed from the <varname>config</varname>. See
|
|
|
|
<literal>lib.systems.parse</literal> for the exact representation.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>libc</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
This is a string identifying the standard C library used. Valid
|
|
|
|
identifiers include "glibc" for GNU libc, "libSystem" for Darwin's
|
|
|
|
Libsystem, and "uclibc" for µClibc. It should probably be refactored to
|
|
|
|
use the module system, like <varname>parse</varname>.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>is*</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
These predicates are defined in <literal>lib.systems.inspect</literal>,
|
2018-11-19 05:10:39 +00:00
|
|
|
and slapped onto every platform. They are superior to the ones in
|
2018-05-01 23:54:21 +00:00
|
|
|
<varname>stdenv</varname> as they force the user to be explicit about
|
|
|
|
which platform they are inspecting. Please use these instead of those.
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
2018-06-01 01:03:37 +00:00
|
|
|
<term>
|
|
|
|
<varname>platform</varname>
|
2018-05-01 23:54:21 +00:00
|
|
|
</term>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
This is, quite frankly, a dumping ground of ad-hoc settings (it's an
|
|
|
|
attribute set). See <literal>lib.systems.platforms</literal> for
|
|
|
|
examples—there's hopefully one in there that will work verbatim for
|
|
|
|
each platform that is working. Please help us triage these flags and
|
|
|
|
give them better homes!
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
2017-01-22 20:52:35 +00:00
|
|
|
</section>
|
|
|
|
|
2019-03-20 22:21:00 +00:00
|
|
|
<section xml:id="ssec-cross-dependency-categorization">
|
|
|
|
<title>Theory of dependency categorization</title>
|
|
|
|
|
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
This is a rather philosophical description that isn't very
|
|
|
|
Nixpkgs-specific. For an overview of all the relevant attributes given to
|
|
|
|
<varname>mkDerivation</varname>, see
|
|
|
|
<xref
|
|
|
|
linkend="ssec-stdenv-dependencies"/>. For a description of how
|
|
|
|
everything is implemented, see
|
|
|
|
<xref linkend="ssec-cross-dependency-implementation" />.
|
|
|
|
</para>
|
|
|
|
</note>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
|
|
|
<para>
|
|
|
|
In this section we explore the relationship between both runtime and
|
2018-11-19 05:10:39 +00:00
|
|
|
build-time dependencies and the 3 Autoconf platforms.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2019-03-20 22:21:00 +00:00
|
|
|
A run time dependency between two packages requires that their host
|
|
|
|
platforms match. This is directly implied by the meaning of "host platform"
|
|
|
|
and "runtime dependency": The package dependency exists while both packages
|
|
|
|
are running on a single host platform.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2019-03-20 22:21:00 +00:00
|
|
|
A build time dependency, however, has a shift in platforms between the
|
|
|
|
depending package and the depended-on package. "build time dependency"
|
|
|
|
means that to build the depending package we need to be able to run the
|
|
|
|
depended-on's package. The depending package's build platform is therefore
|
|
|
|
equal to the depended-on package's host platform.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2019-03-20 22:21:00 +00:00
|
|
|
If both the dependency and depending packages aren't compilers or other
|
|
|
|
machine-code-producing tools, we're done. And indeed
|
|
|
|
<varname>buildInputs</varname> and <varname>nativeBuildInputs</varname>
|
|
|
|
have covered these simpler build-time and run-time (respectively) changes
|
|
|
|
for many years. But if the depedency does produce machine code, we might
|
|
|
|
need to worry about it's target platform too. In principle, that target
|
|
|
|
platform might be any of the depending package's build, host, or target
|
|
|
|
platforms, but we prohibit dependencies from a "later" platform to an
|
|
|
|
earlier platform to limit confusion because we've never seen a legitimate
|
|
|
|
use for them.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2019-03-20 22:21:00 +00:00
|
|
|
Finally, if the depending package is a compiler or other
|
|
|
|
machine-code-producing tool, it might need dependencies that run at "emit
|
|
|
|
time". This is for compilers that (regrettably) insist on being in built
|
|
|
|
together with their source langauges' standard libraries. Assuming build !=
|
|
|
|
host != target, a run-time dependency of the standard library cannot be run
|
|
|
|
at the compiler's build time or run time, but only at the run time of code
|
|
|
|
emitted by the compiler.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2019-03-20 22:21:00 +00:00
|
|
|
Putting this all together, that means we have dependencies in the form
|
|
|
|
"host → target", in at most the following six combinations:
|
|
|
|
<table>
|
|
|
|
<caption>Possible dependency types</caption>
|
|
|
|
<thead>
|
|
|
|
<tr>
|
|
|
|
<th>Dependency's host platform</th>
|
|
|
|
<th>Dependency's target platform</th>
|
|
|
|
</tr>
|
|
|
|
</thead>
|
|
|
|
<tbody>
|
|
|
|
<tr>
|
|
|
|
<td>build</td>
|
|
|
|
<td>build</td>
|
|
|
|
</tr>
|
|
|
|
<tr>
|
|
|
|
<td>build</td>
|
|
|
|
<td>host</td>
|
|
|
|
</tr>
|
|
|
|
<tr>
|
|
|
|
<td>build</td>
|
|
|
|
<td>target</td>
|
|
|
|
</tr>
|
|
|
|
<tr>
|
|
|
|
<td>host</td>
|
|
|
|
<td>host</td>
|
|
|
|
</tr>
|
|
|
|
<tr>
|
|
|
|
<td>host</td>
|
|
|
|
<td>target</td>
|
|
|
|
</tr>
|
|
|
|
<tr>
|
|
|
|
<td>target</td>
|
|
|
|
<td>target</td>
|
|
|
|
</tr>
|
|
|
|
</tbody>
|
|
|
|
</table>
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
|
|
|
|
2019-03-20 22:21:00 +00:00
|
|
|
<para>
|
|
|
|
Some examples will make this table clearer. Suppose there's some package
|
|
|
|
that is being built with a <literal>(build, host, target)</literal>
|
|
|
|
platform triple of <literal>(foo, bar, baz)</literal>. If it has a
|
|
|
|
build-time library dependency, that would be a "host → build" dependency
|
|
|
|
with a triple of <literal>(foo, foo, *)</literal> (the target platform is
|
|
|
|
irrelevant). If it needs a compiler to be built, that would be a "build →
|
|
|
|
host" dependency with a triple of <literal>(foo, foo, *)</literal> (the
|
|
|
|
target platform is irrelevant). That compiler, would be built with another
|
|
|
|
compiler, also "build → host" dependency, with a triple of <literal>(foo,
|
|
|
|
foo, foo)</literal>.
|
|
|
|
</para>
|
2017-01-22 20:52:35 +00:00
|
|
|
</section>
|
|
|
|
|
2019-03-20 22:21:00 +00:00
|
|
|
<section xml:id="ssec-cross-cookbook">
|
2018-07-06 15:52:19 +00:00
|
|
|
<title>Cross packaging cookbook</title>
|
2017-01-22 20:52:35 +00:00
|
|
|
|
2018-05-01 23:54:21 +00:00
|
|
|
<para>
|
2018-11-19 05:23:22 +00:00
|
|
|
Some frequently encountered problems when packaging for cross-compilation
|
|
|
|
should be answered here. Ideally, the information above is exhaustive, so
|
|
|
|
this section cannot provide any new information, but it is ludicrous and
|
|
|
|
cruel to expect everyone to spend effort working through the interaction of
|
2019-03-09 05:07:11 +00:00
|
|
|
many features just to figure out the same answer to the same common
|
|
|
|
problem. Feel free to add to this list!
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
2017-01-22 20:52:35 +00:00
|
|
|
|
2018-05-01 23:54:21 +00:00
|
|
|
<qandaset>
|
2018-09-01 19:20:57 +00:00
|
|
|
<qandaentry xml:id="cross-qa-build-c-program-in-build-environment">
|
2018-05-01 23:54:21 +00:00
|
|
|
<question>
|
|
|
|
<para>
|
|
|
|
What if my package's build system needs to build a C program to be run
|
|
|
|
under the build environment?
|
|
|
|
</para>
|
|
|
|
</question>
|
|
|
|
<answer>
|
|
|
|
<para>
|
|
|
|
<programlisting>depsBuildBuild = [ buildPackages.stdenv.cc ];</programlisting>
|
|
|
|
Add it to your <function>mkDerivation</function> invocation.
|
|
|
|
</para>
|
|
|
|
</answer>
|
|
|
|
</qandaentry>
|
2018-09-01 19:20:57 +00:00
|
|
|
<qandaentry xml:id="cross-qa-fails-to-find-ar">
|
2018-05-01 23:54:21 +00:00
|
|
|
<question>
|
|
|
|
<para>
|
|
|
|
My package fails to find <command>ar</command>.
|
|
|
|
</para>
|
|
|
|
</question>
|
|
|
|
<answer>
|
|
|
|
<para>
|
|
|
|
Many packages assume that an unprefixed <command>ar</command> is
|
|
|
|
available, but Nix doesn't provide one. It only provides a prefixed one,
|
|
|
|
just as it only does for all the other binutils programs. It may be
|
|
|
|
necessary to patch the package to fix the build system to use a prefixed
|
|
|
|
`ar`.
|
|
|
|
</para>
|
|
|
|
</answer>
|
|
|
|
</qandaentry>
|
2018-09-01 19:20:57 +00:00
|
|
|
<qandaentry xml:id="cross-testsuite-runs-host-code">
|
2018-05-01 23:54:21 +00:00
|
|
|
<question>
|
|
|
|
<para>
|
|
|
|
My package's testsuite needs to run host platform code.
|
|
|
|
</para>
|
|
|
|
</question>
|
|
|
|
<answer>
|
|
|
|
<para>
|
|
|
|
<programlisting>doCheck = stdenv.hostPlatform != stdenv.buildPlatfrom;</programlisting>
|
|
|
|
Add it to your <function>mkDerivation</function> invocation.
|
|
|
|
</para>
|
|
|
|
</answer>
|
|
|
|
</qandaentry>
|
|
|
|
</qandaset>
|
|
|
|
</section>
|
|
|
|
</section>
|
|
|
|
<!--============================================================-->
|
|
|
|
<section xml:id="sec-cross-usage">
|
2017-01-22 20:52:35 +00:00
|
|
|
<title>Cross-building packages</title>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
2017-02-08 21:43:52 +00:00
|
|
|
<para>
|
2018-05-01 23:54:21 +00:00
|
|
|
Nixpkgs can be instantiated with <varname>localSystem</varname> alone, in
|
2018-11-19 05:23:22 +00:00
|
|
|
which case there is no cross-compiling and everything is built by and for
|
2018-05-01 23:54:21 +00:00
|
|
|
that system, or also with <varname>crossSystem</varname>, in which case
|
|
|
|
packages run on the latter, but all building happens on the former. Both
|
|
|
|
parameters take the same schema as the 3 (build, host, and target) platforms
|
|
|
|
defined in the previous section. As mentioned above,
|
|
|
|
<literal>lib.systems.examples</literal> has some platforms which are used as
|
|
|
|
arguments for these parameters in practice. You can use them
|
|
|
|
programmatically, or on the command line:
|
|
|
|
<programlisting>
|
2018-01-25 23:32:18 +00:00
|
|
|
nix-build <nixpkgs> --arg crossSystem '(import <nixpkgs/lib>).systems.examples.fooBarBaz' -A whatever</programlisting>
|
2017-02-08 21:43:52 +00:00
|
|
|
</para>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
2018-01-26 17:30:05 +00:00
|
|
|
<note>
|
2018-05-01 23:54:21 +00:00
|
|
|
<para>
|
|
|
|
Eventually we would like to make these platform examples an unnecessary
|
|
|
|
convenience so that
|
|
|
|
<programlisting>
|
2018-11-28 22:00:43 +00:00
|
|
|
nix-build <nixpkgs> --arg crossSystem '{ config = "<arch>-<os>-<vendor>-<abi>"; }' -A whatever</programlisting>
|
2018-05-01 23:54:21 +00:00
|
|
|
works in the vast majority of cases. The problem today is dependencies on
|
|
|
|
other sorts of configuration which aren't given proper defaults. We rely on
|
|
|
|
the examples to crudely to set those configuration parameters in some
|
|
|
|
vaguely sane manner on the users behalf. Issue
|
|
|
|
<link xlink:href="https://github.com/NixOS/nixpkgs/issues/34274">#34274</link>
|
|
|
|
tracks this inconvenience along with its root cause in crufty configuration
|
|
|
|
options.
|
|
|
|
</para>
|
2018-01-26 17:30:05 +00:00
|
|
|
</note>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
2017-05-23 22:04:15 +00:00
|
|
|
<para>
|
2018-05-01 23:54:21 +00:00
|
|
|
While one is free to pass both parameters in full, there's a lot of logic to
|
|
|
|
fill in missing fields. As discussed in the previous section, only one of
|
|
|
|
<varname>system</varname>, <varname>config</varname>, and
|
|
|
|
<varname>parsed</varname> is needed to infer the other two. Additionally,
|
|
|
|
<varname>libc</varname> will be inferred from <varname>parse</varname>.
|
|
|
|
Finally, <literal>localSystem.system</literal> is also
|
|
|
|
<emphasis>impurely</emphasis> inferred based on the platform evaluation
|
|
|
|
occurs. This means it is often not necessary to pass
|
|
|
|
<varname>localSystem</varname> at all, as in the command-line example in the
|
|
|
|
previous paragraph.
|
2017-05-23 22:04:15 +00:00
|
|
|
</para>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
2017-05-23 22:04:15 +00:00
|
|
|
<note>
|
2018-05-01 23:54:21 +00:00
|
|
|
<para>
|
|
|
|
Many sources (manual, wiki, etc) probably mention passing
|
|
|
|
<varname>system</varname>, <varname>platform</varname>, along with the
|
|
|
|
optional <varname>crossSystem</varname> to nixpkgs: <literal>import
|
|
|
|
<nixpkgs> { system = ..; platform = ..; crossSystem = ..;
|
|
|
|
}</literal>. Passing those two instead of <varname>localSystem</varname> is
|
|
|
|
still supported for compatibility, but is discouraged. Indeed, much of the
|
|
|
|
inference we do for these parameters is motivated by compatibility as much
|
|
|
|
as convenience.
|
|
|
|
</para>
|
2017-05-23 22:04:15 +00:00
|
|
|
</note>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
2017-01-22 20:52:35 +00:00
|
|
|
<para>
|
2018-05-01 23:54:21 +00:00
|
|
|
One would think that <varname>localSystem</varname> and
|
|
|
|
<varname>crossSystem</varname> overlap horribly with the three
|
|
|
|
<varname>*Platforms</varname> (<varname>buildPlatform</varname>,
|
|
|
|
<varname>hostPlatform,</varname> and <varname>targetPlatform</varname>; see
|
|
|
|
<varname>stage.nix</varname> or the manual). Actually, those identifiers are
|
|
|
|
purposefully not used here to draw a subtle but important distinction: While
|
|
|
|
the granularity of having 3 platforms is necessary to properly *build*
|
|
|
|
packages, it is overkill for specifying the user's *intent* when making a
|
|
|
|
build plan or package set. A simple "build vs deploy" dichotomy is adequate:
|
|
|
|
the sliding window principle described in the previous section shows how to
|
|
|
|
interpolate between the these two "end points" to get the 3 platform triple
|
2019-03-09 05:07:11 +00:00
|
|
|
for each bootstrapping stage. That means for any package a given package
|
|
|
|
set, even those not bound on the top level but only reachable via
|
|
|
|
dependencies or <varname>buildPackages</varname>, the three platforms will
|
|
|
|
be defined as one of <varname>localSystem</varname> or
|
|
|
|
<varname>crossSystem</varname>, with the former replacing the latter as one
|
|
|
|
traverses build-time dependencies. A last simple difference is that
|
|
|
|
<varname>crossSystem</varname> should be null when one doesn't want to
|
|
|
|
cross-compile, while the <varname>*Platform</varname>s are always non-null.
|
|
|
|
<varname>localSystem</varname> is always non-null.
|
2017-01-22 20:52:35 +00:00
|
|
|
</para>
|
2018-05-01 23:54:21 +00:00
|
|
|
</section>
|
2017-01-22 20:52:35 +00:00
|
|
|
<!--============================================================-->
|
2018-05-01 23:54:21 +00:00
|
|
|
<section xml:id="sec-cross-infra">
|
2017-01-22 20:52:35 +00:00
|
|
|
<title>Cross-compilation infrastructure</title>
|
|
|
|
|
2019-03-20 22:21:00 +00:00
|
|
|
<section xml:id="ssec-cross-dependency-implementation">
|
|
|
|
<title>Implementation of dependencies</title>
|
2018-05-01 23:54:21 +00:00
|
|
|
|
|
|
|
<para>
|
2019-03-20 22:21:00 +00:00
|
|
|
The categorizes of dependencies developed in
|
|
|
|
<xref
|
|
|
|
linkend="ssec-cross-dependency-categorization"/> are specified as
|
|
|
|
lists of derivations given to <varname>mkDerivation</varname>, as
|
|
|
|
documented in <xref linkend="ssec-stdenv-dependencies"/>. In short, the
|
|
|
|
each list of dependencies for "host → target" of "foo → bar" is called
|
|
|
|
<varname>depsFooBar</varname>, with the exceptions for backwards
|
|
|
|
compatibility that <varname>depsBuildHost</varname> is instead called
|
|
|
|
<varname>nativeBuildInputs</varname> and <varname>depsHostTarget</varname>
|
|
|
|
is instead called <varname>buildInputs</varname>. Nixpkgs is now structured
|
|
|
|
so that each <varname>depsFooBar</varname> is automatically taken from
|
|
|
|
<varname>pkgsFooBar</varname>. (These <varname>pkgsFooBar</varname>s are
|
|
|
|
quite new, so there is no special case for
|
|
|
|
<varname>nativeBuildInputs</varname> and <varname>buildInputs</varname>.)
|
|
|
|
For example, <varname>pkgsBuildHost.gcc</varname> should be used at
|
|
|
|
build-time, while <varname>pkgsHostTarget.gcc</varname> should be used at
|
|
|
|
run-time.
|
2018-05-01 23:54:21 +00:00
|
|
|
</para>
|
2019-03-20 22:21:00 +00:00
|
|
|
|
|
|
|
<para>
|
|
|
|
Now, for most of Nixpkgs's history, there was no
|
|
|
|
<varname>pkgsFooBar</varname> attributes, and most packages have not been
|
|
|
|
refactored to use it explicitly. Prior to those, there were just
|
|
|
|
<varname>buildPackages</varname>, <varname>pkgs</varname>, and
|
|
|
|
<varname>targetPackages</varname>. Those are now redefined as aliases to
|
|
|
|
<varname>pkgsBuildHost</varname>, <varname>pkgsHostTarget</varname>, and
|
|
|
|
<varname>pkgsTargetTarget</varname>. It is fine, indeed if anything
|
|
|
|
recommended, to use them for libraries to show that the host platform is
|
|
|
|
irrelevant.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
But before that, there was just <varname>pkgs</varname>, even though both
|
|
|
|
<varname>buildInputs</varname> and <varname>nativeBuildInputs</varname>
|
|
|
|
existed. [Cross barely worked, and those were implemented with some hacks
|
|
|
|
on <varname>mkDerivation</varname> to override dependencies.] What this
|
|
|
|
means is the vast majority of packages do not use any explicit package set
|
|
|
|
to populate their dependencies, just using whatever
|
|
|
|
<varname>callPackage</varname> gives them even if they do correctly sort
|
|
|
|
their dependencies into the multiple lists described above. And indeed,
|
|
|
|
asking that users both sort their dependencies, <emphasis>and</emphasis>
|
|
|
|
take them from the right attribute set, is both too onerous and redundant,
|
|
|
|
so the recommend approach (for now) is to continue just categorizing by
|
|
|
|
list and not using an explicit package set.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
No make this work, we "splice" together the six
|
|
|
|
<varname>pkgsFooBar</varname> package sets and have
|
|
|
|
<varname>callPackage</varname> actually take its arguments from that. This
|
|
|
|
is currently implemented in <filename>pkgs/top-level/splice.nix</filename>.
|
|
|
|
<varname>mkDerivation</varname> then, for each dependency attribute, pulls
|
|
|
|
the right derivation out from the splice. This splicing can be skipped when
|
|
|
|
not cross-compiling as the package sets are the same, but still is a bit
|
|
|
|
slow for cross-compiling. We'd like to do something better, but haven't
|
|
|
|
come up with anything yet.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section xml:id="ssec-bootstrapping">
|
|
|
|
<title>Bootstrapping</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Each of the package sets described above come from a single bootstrapping
|
|
|
|
stage. While <filename>pkgs/top-level/default.nix</filename>, coordinates
|
|
|
|
the composition of stages at a high level,
|
|
|
|
<filename>pkgs/top-level/stage.nix</filename> "ties the knot" (creates the
|
|
|
|
fixed point) of each stage. The package sets are defined per-stage however,
|
|
|
|
so they can be thought of as edges between stages (the nodes) in a graph.
|
|
|
|
Compositions like <literal>pkgsBuildTarget.TargetPackages</literal> can be
|
|
|
|
thought of as paths to this graph.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
While there are many package sets, and thus many edges, the stages can also
|
|
|
|
be arranged in a linear chain. In other words, many of the edges are
|
|
|
|
redundant as far as connectivity is concerned. This hinges on the type of
|
|
|
|
bootstrapping we do. Currently for cross it is:
|
|
|
|
<orderedlist>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
<literal>(native, native, native)</literal>
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
<literal>(native, native, foreign)</literal>
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
<listitem>
|
|
|
|
<para>
|
|
|
|
<literal>(native, foreign, foreign)</literal>
|
|
|
|
</para>
|
|
|
|
</listitem>
|
|
|
|
</orderedlist>
|
|
|
|
In each stage, <varname>pkgsBuildHost</varname> refers the the previous
|
|
|
|
stage, <varname>pkgsBuildBuild</varname> refers to the one before that, and
|
|
|
|
<varname>pkgsHostTarget</varname> refers to the current one, and
|
|
|
|
<varname>pkgsTargetTarget</varname> refers to the next one. When there is
|
|
|
|
no previous or next stage, they instead refer to the current stage. Note
|
|
|
|
how all the invariants about the mapping between dependency and depending
|
|
|
|
packages' build host and target platforms are preserved.
|
|
|
|
<varname>pkgsBuildTarget</varname> and <varname>pkgsHostHost</varname> are
|
|
|
|
more complex in that the stage fitting the requirements isn't always a
|
|
|
|
fixed chain of "prevs" and "nexts" away (modulo the "saturating"
|
|
|
|
self-references at the ends). We just special case instead. All the primary
|
|
|
|
edges are implemented is in <filename>pkgs/stdenv/booter.nix</filename>,
|
|
|
|
and secondarily aliases in <filename>pkgs/top-level/stage.nix</filename>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
Note the native stages are bootstrapped in legacy ways that predate the
|
|
|
|
current cross implementation. This is why the the bootstrapping stages
|
|
|
|
leading up to the final stages are ignored inthe previous paragraph.
|
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If one looks at the 3 platform triples, one can see that they overlap such
|
|
|
|
that one could put them together into a chain like:
|
|
|
|
<programlisting>
|
|
|
|
(native, native, native, foreign, foreign)
|
|
|
|
</programlisting>
|
|
|
|
If one imagines the saturating self references at the end being replaced
|
|
|
|
with infinite stages, and then overlays those platform triples, one ends up
|
|
|
|
with the infinite tuple:
|
|
|
|
<programlisting>
|
|
|
|
(native..., native, native, native, foreign, foreign, foreign...)
|
|
|
|
</programlisting>
|
|
|
|
On can then imagine any sequence of platforms such that there are bootstrap
|
|
|
|
stages with their 3 platforms determined by "sliding a window" that is the
|
|
|
|
3 tuple through the sequence. This was the original model for
|
|
|
|
bootstrapping. Without a target platform (assume a better world where all
|
|
|
|
compilers are multi-target and all standard libraries are built in their
|
|
|
|
own derivation), this is sufficient. Conversely if one wishes to cross
|
|
|
|
compile "faster", with a "Canadian Cross" bootstraping stage where
|
|
|
|
<literal>build != host != target</literal>, more bootstrapping stages are
|
|
|
|
needed since no sliding window providess the pesky
|
|
|
|
<varname>pkgsBuildTarget</varname> package set since it skips the Canadian
|
|
|
|
cross stage's "host".
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
It is much better to refer to <varname>buildPackages</varname> than
|
|
|
|
<varname>targetPackages</varname>, or more broadly package sets that do
|
|
|
|
not mention "target". There are three reasons for this.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
First, it is because bootstrapping stages do not have a unique
|
|
|
|
<varname>targetPackages</varname>. For example a <literal>(x86-linux,
|
|
|
|
x86-linux, arm-linux)</literal> and <literal>(x86-linux, x86-linux,
|
|
|
|
x86-windows)</literal> package set both have a <literal>(x86-linux,
|
|
|
|
x86-linux, x86-linux)</literal> package set. Because there is no canonical
|
|
|
|
<varname>targetPackages</varname> for such a native (<literal>build ==
|
|
|
|
host == target</literal>) package set, we set their
|
|
|
|
<varname>targetPackages</varname>
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Second, it is because this is a frequent source of hard-to-follow
|
|
|
|
"infinite recursions" / cycles. When only packages sets that don't mention
|
|
|
|
target are used, the package set forms a directly acyclic graph. This
|
|
|
|
means that all cycles that exist are confirmed to one stage. This means
|
|
|
|
they are a lot smaller, so easier to follow in the code or a backtrace. It
|
|
|
|
also means they are present in native and cross builds alike, and so more
|
|
|
|
likely to be caught by CI and other users.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Thirdly, it is because everything target-mentioning only exists to
|
|
|
|
accommodate compilers with lousy build systems that insist on the compiler
|
|
|
|
itself and standard library being built together. Of course that is bad
|
|
|
|
because bigger derivation means longer rebuilds. It is also subpar because
|
|
|
|
it tends to make the standard libraries less like other libraries than
|
|
|
|
they could be, complicating code and build systems alike. Because of the
|
|
|
|
other problems, and because of these innate disadvantages, compilers ought
|
|
|
|
to be packaged another way where possible.
|
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
|
|
|
|
<note>
|
|
|
|
<para>
|
|
|
|
If one explores Nixpkgs, they will see derivations with names like
|
|
|
|
<literal>gccCross</literal>. Such <literal>*Cross</literal> derivations is
|
|
|
|
a holdover from before we properly distinguished between the host and
|
|
|
|
target platforms—the derivation with "Cross" in the name covered the
|
|
|
|
<literal>build = host != target</literal> case, while the other covered
|
|
|
|
the <literal>host = target</literal>, with build platform the same or not
|
|
|
|
based on whether one was using its <literal>.nativeDrv</literal> or
|
|
|
|
<literal>.crossDrv</literal>. This ugliness will disappear soon.
|
|
|
|
</para>
|
|
|
|
</note>
|
|
|
|
</section>
|
2018-05-01 23:54:21 +00:00
|
|
|
</section>
|
2017-01-22 20:52:35 +00:00
|
|
|
</chapter>
|