Allow customisation of the algorithm used to convert nix references
graph (created from docker image contents) to docker layers.
A collection of building blocks (python functions) is provided, which
use can assembled into a processing pipeline by specifying a list of
operations (and their initial arguments) via a nix list.
Nix references graph if first converted into a python igraph.Graph
object (with each vertex representing a nix path), which is then fed
into the user defined pipeline. Each stage in the pipeline represents a
function call, with initial arguments specified by the user in nix, and
the last argument being the result of the previous stage in the pipeline
(or the initial Graph object). Each step of the pipeline is expected to
produce a data structure consisting of arbitrarily nested lists/dicts
with Graph objects (representing docker layers) at it's leafs. The
result of the last stage in the pipeline is recursively flattened (with
each dict converted into list of values), until a flat list of Graphs
remains. This is then output as a json array of arrays (each Graph
converted into an array of paths).
This functionality is made available via new `layeringPipeline` argument
to the `streamLayeredImage`/`buildLayeredImage` functions. The default
value of the argument has been chosen to to preserve current layering
behaviour.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
Setting the image creation timestamp in the image metadata to a
constant date can cause problems with self-hosted container
registries, that need to e.g. prune old images. This timestamp is
also useful for debugging.
However, it is almost never useful to set the filesystem timestamp to
a constant value. Doing so not only causes the image to possibly no
longer be reproducible, but also removes any possibility of
deduplicating layers with other images, causing unnecessary storage
space usage.
Therefore, this commit introduces "mtime", a new parameter to
streamLayeredImage, which allows specifying the filesystem timestamps
separately from "created". For backwards compatibility, "mtime"
defaults to the value of "created".
In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
This adds a new `imageStream` option that can be used in conjunction
with `pkgs.dockerTools.streamLayeredImage` so that the image archive
never needs to be materialized in the `/nix/store`. This greatly
improves the disk utilization for systems that use container images
built using Nix because they only need to store image layers instead of
the full image. Additionally, when deploying the new system and only
new layers need to be built/copied.
* streamLayeredImage: self-document the script
'podman load' doesn't let me override the name/tag from the image.
This name and tag is dynamic and in a CI environment so I would like to
be able to adjust the tag dynamically.
Since the image is streamed by stream_layered_image.py, there is no need
to stick with the nix-hardcoded image name/tag: the python script can
accept another name
I've added argparse to expose the feature. It has the added benefit of
adding `--help` support to the script, which makes its usage
self-explanatory.
* blackified file
This removes redundant inline docs, because
- users should consult the better docs in the manual,
- contributors should add to the manual, not the inline comments
This is convenient for debugging the underlying streamed image used by
`dockerTools.buildLayeredImage`.
Here's an example of how you might use this:
```console
$ nix repl ./.
nix-repl> dockerTools.examples.nginx.passthru.stream
«derivation /nix/store/9zczmlp2kraszx4ssmh6fawnlnsa5a4n-stream-nginx-container.drv»
```
The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
The change is insignificant when the owner is root. However, when it
is not root, this change is needed to allow using Nix (as an
unprivileged user) inside the container.
The loaded database contains timestamps of when the nix paths were
registered. Depending on the host store, these can differ between runs.
Resetting them to a well known values ensures that the produced image is
reproducible.
This commit adds support for swapping out the compression algorithm
used in all major docker-tools commands that generate images. The
default algorithm remains unchanged (gzip).
After #268458, when setting `enableFakechroot = true` and
`includeStorePaths = false`, some of the store paths were getting
included into the image anyway, thru `bind-paths`.
This resulted in unexpectedly large images.
Now, the images will not contain any store paths under those
circumstances.
Not sure how this ever worked but tar was trying to archive /proc and /sys, which failed to work.
Since this is never useful for containers to do, we exclude this now in the proot case.
Also fakeroot is not needed when proot is used as it provideds the same feature.
We now cleanly seperate those cases as both are kind of hacks and it's more likely
that the combination will just trigger new bugs.
The command `fakechroot` errored with buffer overflows. The `proot`
command doesn't seem to suffer from the same problem. The tar command
creating the layer errors with "permission denied" on a bunch of paths
in /proc but the layer seems to get built anyway.
Since coreutils v9.2 the `--no-clobber` flag results in a non-zero exit
code when the destination files exist. Using `--update=none` will now
reproduce the old behavior of `--no-clobber`.
However, the `--update=none` flag was introduced in coreutils v9.3 and
thus `mergeImages` will fail if you have an older version than v9.3 in
stdenv after applying this commit.
[coreutils v9.3 changelog](f386722dc0/NEWS (L48))
Without this change, the `--os` and `--arch` switches are disregarded
for operations involving `skopeo inspect` invocations. This means that,
for example, one cannot fetch Linux images while on macOS.
