The chroot environment under mnt had /dev and /sys via bind mounts,
but nothing setting up /proc. The `--mount-proc` argument to unshare
defaults to /proc, which is outside of the chroot envirnoment.
The image tag can be specified or generated from the output hash.
Previously, a generated tag could be recovered from the evaluated
image with some string operations.
However, with the introduction of streamLayeredImage, it's not
feasible to compute the generated tag yourself.
With this change, the imageTag attribute is set unconditionally,
for the buildImage, buildLayeredImage, streamLayeredImage functions.
In https://github.com/NixOS/nixpkgs/pull/58431 the authors ensured that
the resulting layer.tar would always list
/nix/
/nix/store/
first to fully comply to the tar spec. Various refactorings later it is only
ensured to create /nix/ but NOT /nix/store anymore. Instead tar transformed
them to /nix/nix and /nix/nix/store.
Calculating the tarsum after creating a layer is inefficient, since
we have to read the tarball we've just written from the disk.
This commit simultaneously calculates the tarsum while creating the
tarball.
Appending to an existing tar archive repeatedly seems to be a quadratic
operation, since tar seems to traverse the existing archive even using
the `-r, --append` flag. This commit avoids that by passing the list of
files to a single tar invocation.
This is useful when buildLayeredImage is called in a generic way
that should allow simple (base) images to be built, which may not
reference any store paths.
Fixes#78744
My previous change broke when there are more packages than the maximum
number of layers. I had assumed that the `store-path-to-layer.sh` was
only ever passed a single store path, but that is not the case if
there are multiple packages going into the final layer. To fix this, we
loop through the paths going into the final layer, appending them to the
tar file and making sure they end up at the right path.
Since a layer is reserved for "customization", the image can not
contains less than 2 layers.
The user gets the following message at evaluation:
nix-instantiate nixos/tests/docker-tools.nix
trace: the maxLayers argument of dockerTools.buildLayeredImage function must be greather than 1 (current value: 1)
Building a docker image with darwin binaries just yields a confusing
error when ran:
standard_init_linux.go:211: exec user process caused "exec format error"
This change prevents people from building such images in the first place
when tar'ing store paths into layered archives when building layered
images, don't use the absolute nix store path so that tar won't complain
if something new is added to the nix store
when building the final docker image, ignore any file changes tar
detects in the layers. they are all immutable and the only thing that
might change is the number of hard links due to store optimization
Before, every docker image had three extra layers:
1. A `closure` layer which is an internal implementation detail of
calculating the closure of the container
2. a `name-config.json` layer which is the images' run-time
configuration, and has no business being *in* the image as a layer.
3. a "bulk-layers" layer which is again and implementation detail
around collecting the image's closure.
None of these layers need to be in the final product.
dockerTools.buildImageWithNixDb: export USER
Changes to Nix user detection (./src/nix-channel/nix-channel.cc#L-166)
cause this function to error. Exporting USER fixes this.
The architecture of an image should default to the architecture for
which that image is being composed or pulled. buildPackages.go.GOARCH is
an easy way to compute that architecture with the correct terminology.
PR #58431 added /nix/store to each layer.tar. However, the timestamp was
not explicitly set while adding /nix and /nix/store to the archive. This
resulted in different SHA256 hashes of layer.tar between image builds.
This change sets time and owner when tar'ing /nix/store.
The layer order was not correct when a parent image was used: parent
image layers were above the new created layer.
This commits simplifies the code related to layer ordering. In
particular, layers in `layer-list` are ordered from bottom-most to
top-most. This is also the order of layers in the `rootfs.diff_ids`
attribute of the image configuration.
Whenever we create scripts that are installed to $out, we must use runtimeShell
in order to get the shell that can be executed on the machine we create the
package for. This is relevant for cross-compiling. The only use case for
stdenv.shell are scripts that are executed as part of the build system.
Usages in checkPhase are borderline however to decrease the likelyhood
of people copying the wrong examples, I decided to use runtimeShell as well.
bcf54ce5bb introduced a treewide change to
use ${stdenv.shell} where-ever possible. However, this broke a script
used by dockerTools, store-path-to-layer.sh, as it did not preserve the
+x mode bit. This meant the file got put into the store as mode 0444,
resulting in a build-time error later on that looked like:
xargs: /nix/store/jixivxhh3c8sncp9xlkc4ls3y5f2mmxh-store-path-to-layer.sh: Permission denied
However, in a twist of fate, bcf54ce5bb
not only introduced this regression but, in this particular instance,
didn't even fix the original bug: the store-path-to-layer.sh script
*still* uses /bin/sh as its shebang line, rather than an absolute path
to stdenv. (Fixing this can be done in a separate commit.)
Signed-off-by: Austin Seipp <aseipp@pobox.com>
This patch preserves the ordering of layers of a parent image when the
new image is packed.
It is currently not the case: layers are stacked in the reverse order.
Fixes#55290
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:
imageA ----> layerA
|
v
layerB
|
v
layerC
The current image spec changed this format to where the Image defined
the order and set of layers:
imageA ---> layerA
|--> layerB
`--> layerC
For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:
imageA ---> layerA
| |
| v
|--> layerB
| |
| v
`--> layerC
This is nice for tooling which supported the older version and never
updated to support the newer format.
Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.
This is not a problem in general, but is a problem with
`buildLayeredImage`.
`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.
This means until now, `buildLayeredImage` could not be the input to
`buildImage`.
The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.
This work has been sponsored by Target.
