Allow customisation of the algorithm used to convert nix references
graph (created from docker image contents) to docker layers.
A collection of building blocks (python functions) is provided, which
use can assembled into a processing pipeline by specifying a list of
operations (and their initial arguments) via a nix list.
Nix references graph if first converted into a python igraph.Graph
object (with each vertex representing a nix path), which is then fed
into the user defined pipeline. Each stage in the pipeline represents a
function call, with initial arguments specified by the user in nix, and
the last argument being the result of the previous stage in the pipeline
(or the initial Graph object). Each step of the pipeline is expected to
produce a data structure consisting of arbitrarily nested lists/dicts
with Graph objects (representing docker layers) at it's leafs. The
result of the last stage in the pipeline is recursively flattened (with
each dict converted into list of values), until a flat list of Graphs
remains. This is then output as a json array of arrays (each Graph
converted into an array of paths).
This functionality is made available via new `layeringPipeline` argument
to the `streamLayeredImage`/`buildLayeredImage` functions. The default
value of the argument has been chosen to to preserve current layering
behaviour.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
We are migrating packages that meet below requirements:
1. using `callPackage`
2. called path is a directory
3. overriding set is empty (`{ }`)
4. not containing path expressions other than relative path (to
makenixpkgs-vet happy)
5. not referenced by nix files outside of the directory, other
than`pkgs/top-level/all-packages.nix`
6. not referencing nix files outside of the directory
7. not referencing `default.nix` (since it's changed to `package.nix`)
8. `outPath` doesn't change after migration
The tool is here: https://github.com/Aleksanaa/by-name-migrate.
This replaces the mess of buildEnvs with a single Rust binary that
spits out a mostly-complete root filesystem for an fhsenv.
The main goal is to have includeClosures, as we want all of the
dependencies to be in the fhsenv to avoid Steam's (and others')
LD_LIBRARY_PATH shenanigans, but without 32-bit libraries leaking
into lib64 when a 64-bit package like mangohud depends on a 32-bit
version of itself.
We "fix" this by actually looking at the files and explicitly moving
32-bit stuff to $out/lib32. This could be avoided if we had recursive
Nix, or at least system info in exportReferencesGraph, but alas.
For some reason this also shrinks the fhsenvs massively, even though
there's currently no layout optimization (e.g. a package with paths
like lib/foo/{bar,baz} will produce two symlinks in the output, even
when it's more optimal to symlink lib/foo to $out/lib/foo directly).
Originally, we switched to bsdtar from libarchive to solve a reproducibility issue related to hardlinks
As of gnu cpio 2.14 the --ignore-dirnlink option is introduced and now included in --reproducible, which solves this issue
By switching back, we are in turn solving an issue in libarchive >=3.7.5 erroring out with "Error reading archive -: (null)"
Change-Id: Ib6140d599b6547d8e941b0251ce996e303c41fa6
Now that ld.so is symlinked into the libc derivation,
we can use it as a dynamic linker.
In my testing, OpenBSD does not have any issues with executing from
a symlinked ld.so.
`fetchNpmDeps` works from the exact same source tree, thus `patchFlags`
must be respected so that any patches get applied correctly.
Signed-off-by: Christoph Heiss <christoph@c8h4.io>
The indentation stripping semantics of strings are fairly bad and have a
few gotchas where the resulting string has not the intended indentation.
This commit fixes most if not all such instances in Nixpkgs.
I tried to strive a balance between keeping the diff small and
reformatting/refactoring the code to look better. In general,
reformatting should be left to Nixfmt.
Note that this causes a lot of rebuilds by design. All changes need to
be thoroughly vetted and reviewed for correctness. There is no automatic
way to prove correctness.
List of files to fix generated by running
https://gerrit.lix.systems/c/lix/+/2092 on Nixpkgs and looking at the
warnings.
This is a follow-up to a169553f7e.
In most cases it should allow the TLS verification to be enabled.
It also makes the behavior of `fetchurl` more consistent with other fetchers
like `fetchgit`.
Ideally we would always fallback on `cacert` but I am not sure how to build
`cacert` during bootstrap without making an unmaintainable mess.
