From 5e5266f83fc2cce2b353601da0f29bd6805d4597 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 20 Mar 2019 18:21:00 -0400 Subject: [PATCH] manual: Document `pkgsFooBar` and more There was a bunch of stuff in the cross section that haddn't had any attention in a while. I might need to slim it down later, but this is good for now. --- doc/cross-compilation.xml | 374 +++++++++++++++++++++++++++++--------- doc/stdenv.xml | 12 +- 2 files changed, 299 insertions(+), 87 deletions(-) diff --git a/doc/cross-compilation.xml b/doc/cross-compilation.xml index dbaf6f104ec0..d97f12f25661 100644 --- a/doc/cross-compilation.xml +++ b/doc/cross-compilation.xml @@ -12,11 +12,12 @@ computing power and memory to compile their own programs. One might think that cross-compilation is a fairly niche concern. However, there are significant advantages to rigorously distinguishing between build-time and - run-time environments! This applies even when one is developing and - deploying on the same machine. Nixpkgs is increasingly adopting the opinion - that packages should be written with cross-compilation in mind, and nixpkgs - should evaluate in a similar way (by minimizing cross-compilation-specific - special cases) whether or not one is cross-compiling. + run-time environments! Significant, because the benefits apply even when one + is developing and deploying on the same machine. Nixpkgs is increasingly + adopting the opinion that packages should be written with cross-compilation + in mind, and nixpkgs should evaluate in a similar way (by minimizing + cross-compilation-specific special cases) whether or not one is + cross-compiling. @@ -30,7 +31,7 @@
Packaging in a cross-friendly manner -
+
Platform parameters @@ -218,8 +219,20 @@
-
- Specifying Dependencies +
+ Theory of dependency categorization + + + + This is a rather philosophical description that isn't very + Nixpkgs-specific. For an overview of all the relevant attributes given to + mkDerivation, see + . For a description of how + everything is implemented, see + . + + In this section we explore the relationship between both runtime and @@ -227,84 +240,98 @@ - A runtime dependency between 2 packages implies that between them both the - host and target platforms match. This is directly implied by the meaning of - "host platform" and "runtime dependency": The package dependency exists - while both packages are running on a single host platform. + A run time dependency between two packages requires that their host + platforms match. This is directly implied by the meaning of "host platform" + and "runtime dependency": The package dependency exists while both packages + are running on a single host platform. - A build time dependency, however, implies a shift in platforms between the - depending package and the depended-on package. The meaning of a build time - dependency is that to build the depending package we need to be able to run - the depended-on's package. The depending package's build platform is - therefore equal to the depended-on package's host platform. Analogously, - the depending package's host platform is equal to the depended-on package's - target platform. + A build time dependency, however, has a shift in platforms between the + depending package and the depended-on package. "build time dependency" + means that to build the depending package we need to be able to run the + depended-on's package. The depending package's build platform is therefore + equal to the depended-on package's host platform. - In this manner, given the 3 platforms for one package, we can determine the - three platforms for all its transitive dependencies. This is the most - important guiding principle behind cross-compilation with Nixpkgs, and will - be called the sliding window principle. + If both the dependency and depending packages aren't compilers or other + machine-code-producing tools, we're done. And indeed + buildInputs and nativeBuildInputs + have covered these simpler build-time and run-time (respectively) changes + for many years. But if the depedency does produce machine code, we might + need to worry about it's target platform too. In principle, that target + platform might be any of the depending package's build, host, or target + platforms, but we prohibit dependencies from a "later" platform to an + earlier platform to limit confusion because we've never seen a legitimate + use for them. - Some examples will make this clearer. If a package is being built with a - (build, host, target) platform triple of (foo, - bar, bar), then its build-time dependencies would have a triple - of (foo, foo, bar), and those - packages' build-time dependencies would have a triple of - (foo, foo, foo). In other words, it should take two - "rounds" of following build-time dependency edges before one reaches a - fixed point where, by the sliding window principle, the platform triple no - longer changes. Indeed, this happens with cross-compilation, where only - rounds of native dependencies starting with the second necessarily coincide - with native packages. + Finally, if the depending package is a compiler or other + machine-code-producing tool, it might need dependencies that run at "emit + time". This is for compilers that (regrettably) insist on being in built + together with their source langauges' standard libraries. Assuming build != + host != target, a run-time dependency of the standard library cannot be run + at the compiler's build time or run time, but only at the run time of code + emitted by the compiler. - - - The depending package's target platform is unconstrained by the sliding - window principle, which makes sense in that one can in principle build - cross compilers targeting arbitrary platforms. - - - - How does this work in practice? Nixpkgs is now structured so that - build-time dependencies are taken from buildPackages, - whereas run-time dependencies are taken from the top level attribute set. - For example, buildPackages.gcc should be used at - build-time, while gcc should be used at run-time. Now, - for most of Nixpkgs's history, there was no - buildPackages, and most packages have not been - refactored to use it explicitly. Instead, one can use the six - (gasp) attributes used for specifying dependencies as - documented in . We "splice" - together the run-time and build-time package sets with - callPackage, and then mkDerivation - for each of four attributes pulls the right derivation out. This splicing - can be skipped when not cross-compiling as the package sets are the same, - but is a bit slow for cross-compiling. Because of this, a - best-of-both-worlds solution is in the works with no splicing or explicit - access of buildPackages needed. For now, feel free to - use either method. + Putting this all together, that means we have dependencies in the form + "host → target", in at most the following six combinations: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Possible dependency types
Dependency's host platformDependency's target platform
buildbuild
buildhost
buildtarget
hosthost
hosttarget
targettarget
 - - - There is also a "backlink" targetPackages, yielding a - package set whose buildPackages is the current package - set. This is a hack, though, to accommodate compilers with lousy build - systems. Please do not use this unless you are absolutely sure you are - packaging such a compiler and there is no other way. - - + + Some examples will make this table clearer. Suppose there's some package + that is being built with a (build, host, target) + platform triple of (foo, bar, baz). If it has a + build-time library dependency, that would be a "host → build" dependency + with a triple of (foo, foo, *) (the target platform is + irrelevant). If it needs a compiler to be built, that would be a "build → + host" dependency with a triple of (foo, foo, *) (the + target platform is irrelevant). That compiler, would be built with another + compiler, also "build → host" dependency, with a triple of (foo, + foo, foo). +
-
+
Cross packaging cookbook @@ -450,21 +477,202 @@ nix-build <nixpkgs> --arg crossSystem '{ config = "<arch>-<os>
Cross-compilation infrastructure - - To be written. - +
+ Implementation of dependencies - - If one explores Nixpkgs, they will see derivations with names like - gccCross. Such *Cross derivations is - a holdover from before we properly distinguished between the host and - target platforms—the derivation with "Cross" in the name covered the - build = host != target case, while the other covered the - host = target, with build platform the same or not based - on whether one was using its .nativeDrv or - .crossDrv. This ugliness will disappear soon. + The categorizes of dependencies developed in + are specified as + lists of derivations given to mkDerivation, as + documented in . In short, the + each list of dependencies for "host → target" of "foo → bar" is called + depsFooBar, with the exceptions for backwards + compatibility that depsBuildHost is instead called + nativeBuildInputs and depsHostTarget + is instead called buildInputs. Nixpkgs is now structured + so that each depsFooBar is automatically taken from + pkgsFooBar. (These pkgsFooBars are + quite new, so there is no special case for + nativeBuildInputs and buildInputs.) + For example, pkgsBuildHost.gcc should be used at + build-time, while pkgsHostTarget.gcc should be used at + run-time. - + + + Now, for most of Nixpkgs's history, there was no + pkgsFooBar attributes, and most packages have not been + refactored to use it explicitly. Prior to those, there were just + buildPackages, pkgs, and + targetPackages. Those are now redefined as aliases to + pkgsBuildHost, pkgsHostTarget, and + pkgsTargetTarget. It is fine, indeed if anything + recommended, to use them for libraries to show that the host platform is + irrelevant. + + + + But before that, there was just pkgs, even though both + buildInputs and nativeBuildInputs + existed. [Cross barely worked, and those were implemented with some hacks + on mkDerivation to override dependencies.] What this + means is the vast majority of packages do not use any explicit package set + to populate their dependencies, just using whatever + callPackage gives them even if they do correctly sort + their dependencies into the multiple lists described above. And indeed, + asking that users both sort their dependencies, and + take them from the right attribute set, is both too onerous and redundant, + so the recommend approach (for now) is to continue just categorizing by + list and not using an explicit package set. + + + + No make this work, we "splice" together the six + pkgsFooBar package sets and have + callPackage actually take its arguments from that. This + is currently implemented in pkgs/top-level/splice.nix. + mkDerivation then, for each dependency attribute, pulls + the right derivation out from the splice. This splicing can be skipped when + not cross-compiling as the package sets are the same, but still is a bit + slow for cross-compiling. We'd like to do something better, but haven't + come up with anything yet. + +
+ +
+ Bootstrapping + + + Each of the package sets described above come from a single bootstrapping + stage. While pkgs/top-level/default.nix, coordinates + the composition of stages at a high level, + pkgs/top-level/stage.nix "ties the knot" (creates the + fixed point) of each stage. The package sets are defined per-stage however, + so they can be thought of as edges between stages (the nodes) in a graph. + Compositions like pkgsBuildTarget.TargetPackages can be + thought of as paths to this graph. + + + + While there are many package sets, and thus many edges, the stages can also + be arranged in a linear chain. In other words, many of the edges are + redundant as far as connectivity is concerned. This hinges on the type of + bootstrapping we do. Currently for cross it is: + + + + (native, native, native) + + + + + (native, native, foreign) + + + + + (native, foreign, foreign) + + + + In each stage, pkgsBuildHost refers the the previous + stage, pkgsBuildBuild refers to the one before that, and + pkgsHostTarget refers to the current one, and + pkgsTargetTarget refers to the next one. When there is + no previous or next stage, they instead refer to the current stage. Note + how all the invariants about the mapping between dependency and depending + packages' build host and target platforms are preserved. + pkgsBuildTarget and pkgsHostHost are + more complex in that the stage fitting the requirements isn't always a + fixed chain of "prevs" and "nexts" away (modulo the "saturating" + self-references at the ends). We just special case instead. All the primary + edges are implemented is in pkgs/stdenv/booter.nix, + and secondarily aliases in pkgs/top-level/stage.nix. + + + + + Note the native stages are bootstrapped in legacy ways that predate the + current cross implementation. This is why the the bootstrapping stages + leading up to the final stages are ignored inthe previous paragraph. + + + + + If one looks at the 3 platform triples, one can see that they overlap such + that one could put them together into a chain like: + +(native, native, native, foreign, foreign) + + If one imagines the saturating self references at the end being replaced + with infinite stages, and then overlays those platform triples, one ends up + with the infinite tuple: + +(native..., native, native, native, foreign, foreign, foreign...) + + On can then imagine any sequence of platforms such that there are bootstrap + stages with their 3 platforms determined by "sliding a window" that is the + 3 tuple through the sequence. This was the original model for + bootstrapping. Without a target platform (assume a better world where all + compilers are multi-target and all standard libraries are built in their + own derivation), this is sufficient. Conversely if one wishes to cross + compile "faster", with a "Canadian Cross" bootstraping stage where + build != host != target, more bootstrapping stages are + needed since no sliding window providess the pesky + pkgsBuildTarget package set since it skips the Canadian + cross stage's "host". + + + + + It is much better to refer to buildPackages than + targetPackages, or more broadly package sets that do + not mention "target". There are three reasons for this. + + + First, it is because bootstrapping stages do not have a unique + targetPackages. For example a (x86-linux, + x86-linux, arm-linux) and (x86-linux, x86-linux, + x86-windows) package set both have a (x86-linux, + x86-linux, x86-linux) package set. Because there is no canonical + targetPackages for such a native (build == + host == target) package set, we set their + targetPackages + + + Second, it is because this is a frequent source of hard-to-follow + "infinite recursions" / cycles. When only packages sets that don't mention + target are used, the package set forms a directly acyclic graph. This + means that all cycles that exist are confirmed to one stage. This means + they are a lot smaller, so easier to follow in the code or a backtrace. It + also means they are present in native and cross builds alike, and so more + likely to be caught by CI and other users. + + + Thirdly, it is because everything target-mentioning only exists to + accommodate compilers with lousy build systems that insist on the compiler + itself and standard library being built together. Of course that is bad + because bigger derivation means longer rebuilds. It is also subpar because + it tends to make the standard libraries less like other libraries than + they could be, complicating code and build systems alike. Because of the + other problems, and because of these innate disadvantages, compilers ought + to be packaged another way where possible. + + + + + + If one explores Nixpkgs, they will see derivations with names like + gccCross. Such *Cross derivations is + a holdover from before we properly distinguished between the host and + target platforms—the derivation with "Cross" in the name covered the + build = host != target case, while the other covered + the host = target, with build platform the same or not + based on whether one was using its .nativeDrv or + .crossDrv. This ugliness will disappear soon. + + +
diff --git a/doc/stdenv.xml b/doc/stdenv.xml index 7e6c589d9fe9..85efbc1dd9dd 100644 --- a/doc/stdenv.xml +++ b/doc/stdenv.xml @@ -222,9 +222,10 @@ genericBuild But even if one is not cross compiling, the platforms imply whether or not the dependency is needed at run-time or build-time, a concept that makes - perfect sense outside of cross compilation. For now, the run-time/build-time - distinction is just a hint for mental clarity, but in the future it perhaps - could be enforced. + perfect sense outside of cross compilation. By default, the + run-time/build-time distinction is just a hint for mental clarity, but with + strictDeps set it is somewhat enforced even in the native + case. @@ -348,7 +349,10 @@ let f(h, h + 1, i) = i + h Overall, the unifying theme here is that propagation shouldn't be introducing transitive dependencies involving platforms the depending - package is unaware of. The offset bounds checking and definition of + package is unaware of. [One can imagine the dependending package asking for + dependencies with the platforms it knows about; other platforms it doesn't + know how to ask for. The platform description in that scenario is a kind of + unforagable capability.] The offset bounds checking and definition of mapOffset together ensure that this is the case. Discovering a new offset is discovering a new platform, and since those platforms weren't in the derivation "spec" of the needing package, they