Commit Graph

468988 Commits

Author SHA1 Message Date
Maximilian Bosch
43dbeae02d
postgresql: pass through JIT-enabled variant of non-JIT postgres and vice versa
This is useful if your postgresql version is dependant on
`system.stateVersion` and not pinned down manually. Then it's not
necessary to find out which version exactly is in use and define
`package` manually, but just stay with what NixOS provides as default:

    $ nix-instantiate -A postgresql
    /nix/store/82fzmb77mz2b787dgj7mn4a8i4f6l6sn-postgresql-14.7.drv
    $ nix-instantiate -A postgresql_jit
    /nix/store/qsjkb72fcrrfpsszrwbsi9q9wgp39m50-postgresql-14.7.drv
    $ nix-instantiate -A postgresql.withJIT
    /nix/store/qsjkb72fcrrfpsszrwbsi9q9wgp39m50-postgresql-14.7.drv
    $ nix-instantiate -A postgresql.withJIT.withoutJIT
    /nix/store/82fzmb77mz2b787dgj7mn4a8i4f6l6sn-postgresql-14.7.drv

I.e. you can use postgresql with JIT (for complex queries only[1]) like
this:

    services.postgresql = {
      enable = true;
      enableJIT = true;
    };

Performing a new override instead of re-using the `_jit`-variants for
that has the nice property that overlays for the original package apply
to the JIT-enabled variant, i.e.

    with import ./. {
      overlays = [
        (self: super: {
          postgresql = super.postgresql.overrideAttrs (_: { fnord = "snens"; });
        })
      ];
    };
    postgresql.withJIT.fnord

still gives the string `snens` whereas `postgresql_jit` doesn't have the
attribute `fnord` in its derivation.

[1] https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-JIT-ABOVE-COST
2023-03-29 08:39:46 +02:00
Maximilian Bosch
2282fa73a1
postgresql: implement opt-in JIT support
Closes #150801

Note: I decided against resuming directly on #150801 because the
conflict was too big (and resolving it seemed too error-prone to me).
Also the `this`-refactoring could be done in an easier manner, i.e. by
exposing JIT attributes with the correct configuration. More on that
below.

This patch creates variants of the `postgresql*`-packages with JIT[1]
support. Please note that a lot of the work was derived from previous
patches filed by other contributors, namely dasJ, andir and abbradar,
hence the co-authored-by tags below.

Effectively, the following things have changed:

* For JIT variants an LLVM-backed stdenv with clang is now used as
  suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the
  JIT-part, so no need to mix this up with GCC's stdenv. Also, using the
  `dev`-output of LLVM and clang's stdenv for building (and adding llvm
  libs as build-inputs) seems more cross friendly to me (which will
  become useful when cross-building for JIT-variants will actually be
  supported).

* Plugins inherit the build flags from the Makefiles in
  `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since
  some of the flags are clang-specific (and stem from the use of the
  CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql`
  is passed to the plugins. I.e., plugins for non-JIT variants are built
  with a gcc stdenv on Linux and plugins for JIT variants with a clang
  stdenv.

  Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it
  as broken for JIT-variants of postgresql only.

* Added a test-matrix to confirm that JIT works fine on each
  `pkgs.postgresql_*_jit` (thanks Andi for the original test in
  #124804!).

* For each postgresql version, a new attribute
  `postgresql_<version>_jit` (and a corresponding
  `postgresqlPackages<version>JitPackages`) are now exposed for better
  discoverability and prebuilt artifacts in the binary cache.

* In #150801 the `this`-argument was replaced by an internal recursion.
  I decided against this approach because it'd blow up the diff even
  more which makes the readability way harder and also harder to revert
  this if necessary.

  Instead, it is made sure that `this` always points to the correct
  variant of `postgresql` and re-using that in an additional
  `.override {}`-expression is trivial because the JIT-variant is
  exposed in `all-packages.nix`.

* I think the changes are sufficiently big to actually add myself as
  maintainer here.

* Added `libxcrypt` to `buildInputs` for versions <v13. While
  building things with an LLVM stdenv, these versions complained that
  the extern `crypt()` symbol can't be found. Not sure what this is
  exactly about, but since we want to switch to libxcrypt for `crypt()`
  usage anyways[5] I decided to add it. For >=13 it's not relevant
  anymore anyways[6].

* JIT support doesn't work with cross-compilation. It is attempted to
  build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for
  each sub-directory in `backend/` for the JIT apparently, but with a
  $(CLANG) that can produce binaries for the build, not the host-platform.

  I managed to get a cross-build with JIT support working with
  `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but
  considering that the resulting LLVM IR isn't platform-independent this
  doesn't give you much. In fact, I tried to test the result in a VM-test,
  but as soon as JIT was used to optimize a query, postgres would
  coredump with `Illegal instruction`.

A common concern of the original approach - with llvm as build input -
was the massive increase of closure size. With the new approach of using
the LLVM stdenv directly and patching out references to the clang drv in
`$out` the effective closure size changes are:

    $ nix path-info -Sh $(nix-build -A postgresql_14)
    /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7	306.4M
    $ nix path-info -Sh $(nix-build -A postgresql_14_jit)
    /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7	689.2M

Most of the increase in closure-size stems from the `lib`-output of
LLVM

    $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib
    /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib	349.8M

which is why this shouldn't be enabled by default.

While this is quite much because of LLVM, it's still a massive
improvement over the simple approach of adding llvm/clang as
build-inputs and building with `--with-llvm`:

    $ nix path-info -Sh $(nix-build -E '
	with import ./. {};
	postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: {
	  configureFlags = configureFlags ++ [ "--with-llvm" ];
	  buildInputs = buildInputs ++ [ llvm clang ];
	})' -j0)
    /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7	  1.6G

Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
Co-authored-by: Janne Heß <janne@hess.ooo>
Co-authored-by: Nikolay Amiantov <ab@fmap.me>

[1] https://www.postgresql.org/docs/current/jit-reason.html
[2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931
    & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321
[3] This fails with the following error otherwise:
    ```
    configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG=
    ```
[4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14
[5] https://github.com/NixOS/nixpkgs/pull/181764
[6] c45643d618
2023-03-29 08:39:46 +02:00
Fabian Affolter
fd9ea56630 python310Packages.gradient: update postPatch section 2023-03-29 08:38:36 +02:00
Jörg Thalheim
3ac65fce5c
Merge pull request #223696 from linj-fork/fix-zshenv-guard-variable
zsh: fix the guard variable for /etc/set-environment
2023-03-29 07:27:20 +01:00
Weijia Wang
dc97ccd73e
Merge pull request #223604 from r-ryantm/auto-update/avalanchego
avalanchego: 1.9.11 -> 1.9.16
2023-03-29 09:26:20 +03:00
Weijia Wang
626401c783
Merge pull request #220832 from wegank/glucose-bump
glucose: 4.1 -> 4.2.1
2023-03-29 09:13:21 +03:00
Fabian Affolter
e68e28ef86 python310Packages.angr: 9.2.43 -> 9.2.44 2023-03-29 08:09:19 +02:00
Fabian Affolter
d7ff296ced python310Packages.cle: 9.2.43 -> 9.2.44 2023-03-29 08:09:18 +02:00
Fabian Affolter
54ba8a534b python310Packages.claripy: 9.2.43 -> 9.2.44 2023-03-29 08:09:18 +02:00
Fabian Affolter
8505db1b37 python310Packages.pyvex: 9.2.43 -> 9.2.44 2023-03-29 08:09:18 +02:00
Fabian Affolter
5c35bc0c4c python310Packages.ailment: 9.2.43 -> 9.2.44 2023-03-29 08:09:18 +02:00
Fabian Affolter
38a8a34575 python310Packages.archinfo: 9.2.43 -> 9.2.44 2023-03-29 08:09:18 +02:00
Jörg Thalheim
8667b0cb36
Merge pull request #223626 from LeSuisse/rizin-0.5.2
rizin: 0.5.1 -> 0.5.2
2023-03-29 07:09:12 +01:00
Weijia Wang
d805a34669
Merge pull request #222868 from r-ryantm/auto-update/netpbm
netpbm: 11.1.0 -> 11.2.0
2023-03-29 09:08:09 +03:00
Fabian Affolter
9fc2969096
Merge pull request #223652 from fabaff/databricks-sql-connector-bump
python310Packages.databricks-sql-connector: mark as broken
2023-03-29 08:05:24 +02:00
Fabian Affolter
babe15e38c
Merge pull request #223653 from fabaff/malduck-broken
python310Packages.malduck: mark as broken
2023-03-29 08:05:07 +02:00
Fabian Affolter
e585ae75eb
Merge pull request #223421 from fabaff/zkg
zkg: init at 2.13.0
2023-03-29 08:03:59 +02:00
Fabian Affolter
e0bcdc1886
Merge pull request #223642 from fabaff/mypy-boto3-s3-bump
python310Packages.mypy-boto3-s3: 1.26.97.post2 -> 1.26.99
2023-03-29 08:03:22 +02:00
Fabian Affolter
e335fe9b00
Merge pull request #223641 from fabaff/mypy-boto3-builder-bump
python310Packages.mypy-boto3-builder: 7.14.2 -> 7.14.4
2023-03-29 08:02:56 +02:00
Fabian Affolter
65b3c9f654
Merge pull request #223643 from fabaff/peaqevcore-bump
python310Packages.peaqevcore: 13.4.1 -> 13.4.2
2023-03-29 08:02:23 +02:00
Fabian Affolter
701b97f1dd
Merge pull request #223621 from fabaff/pychromecast-bump
python310Packages.pychromecast: 13.0.4 -> 13.0.6
2023-03-29 08:01:35 +02:00
Fabian Affolter
fc67b4a1e4
Merge pull request #223637 from fabaff/ytmusicapi-bump
python310Packages.ytmusicapi: 0.25.0 -> 0.25.1
2023-03-29 08:01:23 +02:00
Fabian Affolter
2d1d6c3f40
Merge pull request #223639 from fabaff/yaramod-bump
python310Packages.yaramod: 3.19.0 -> 3.19.1
2023-03-29 08:01:01 +02:00
Lin Jian
43f45fe24f
zsh: fix the guard variable for /etc/set-environment
This guard variable is written to $out/etc/zshenv using
here-documents, so we need to escape it.
2023-03-29 13:51:26 +08:00
Peter Simons
7942e2e38f
Merge pull request #221108 from mrobbetts/bind_remove_allow_query
bind: remove hard-coded `allow-query` config setting
2023-03-29 07:50:47 +02:00
R. Ryantm
7c69b2bbae python310Packages.google-cloud-bigquery-storage: 2.19.0 -> 2.19.1 2023-03-29 05:07:07 +00:00
José Romildo
4929fad3c7 liferea: 1.14.2 -> 1.14.3 2023-03-29 07:04:59 +02:00
R. Ryantm
f0940dac89 fastly: 8.1.0 -> 8.1.2 2023-03-29 04:14:39 +00:00
xinyangli
58f88f61b3 jetbrains: add libxcrypt-legacy 2023-03-29 12:09:24 +08:00
github-actions[bot]
777a7125e8 terraform-providers.snowflake: 0.59.0 → 0.60.0 2023-03-29 14:06:49 +10:00
github-actions[bot]
e4a8b3f5bf terraform-providers.newrelic: 3.18.1 → 3.19.0 2023-03-29 14:06:49 +10:00
github-actions[bot]
07b267273a terraform-providers.hcloud: 1.36.2 → 1.37.0 2023-03-29 14:06:49 +10:00
github-actions[bot]
c9480b5eb2 terraform-providers.google-beta: 4.58.0 → 4.59.0 2023-03-29 14:06:49 +10:00
github-actions[bot]
c7bbb4cddb terraform-providers.google: 4.58.0 → 4.59.0 2023-03-29 14:06:49 +10:00
github-actions[bot]
a23cbedae8 terraform-providers.buildkite: 0.12.1 → 0.12.2 2023-03-29 14:06:49 +10:00
zowoq
ece0980794 clusterctl: 1.3.5 -> 1.4.0
Diff: https://github.com/kubernetes-sigs/cluster-api/compare/v1.3.5...v1.4.0

Changelog: https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.4.0
2023-03-29 13:54:17 +10:00
R. Ryantm
2e5274f934 golangci-lint-langserver: 0.0.7 -> 0.0.8 2023-03-29 02:25:59 +00:00
Orivej Desh
4a6f9a7bd6
raysession: init at 0.13.1 (#222628)
Fixes #194022
2023-03-29 01:54:51 +00:00
Orivej Desh
ee0efdd6a0
patchance: init at 1.0.0 (#222623)
Fixes #194023
2023-03-29 01:54:21 +00:00
Orivej Desh
1ff621cf56
tonelib-noisereducer: init at 1.2.0 (#222124) 2023-03-29 01:53:49 +00:00
Nick Cao
626a6ffeee
Merge pull request #223574 from figsoda/star-history
star-history: 1.0.10 -> 1.0.11
2023-03-29 09:28:39 +08:00
Nick Cao
a687d8f03c
Merge pull request #223576 from figsoda/cargo-tally
cargo-tally: 1.0.23 -> 1.0.24
2023-03-29 09:28:11 +08:00
Nick Cao
0f510e7413
Merge pull request #223526 from kilianar/logseq-0.9.0
logseq: 0.8.18 -> 0.9.0
2023-03-29 09:26:00 +08:00
Nick Cao
aa539367a8
Merge pull request #223539 from xrelkd/update/fcitx5
fcitx5: 5.0.22 -> 5.0.23
2023-03-29 09:25:25 +08:00
Nick Cao
70e7205a96
Merge pull request #223564 from mweinelt/pyoctoprintapi
python310Packages.pyoctoprintapi: 0.1.11 -> 0.1.12
2023-03-29 09:25:04 +08:00
Nick Cao
b54ee48580
Merge pull request #223461 from newAM/svdtools-0.3.0
svdtools: 0.2.8 -> 0.3.0
2023-03-29 09:23:45 +08:00
Nick Cao
622d54b8c8
Merge pull request #223597 from lionello/pulumi-bin-3.60
pulumi-bin: 3.56.0 -> 3.60.0
2023-03-29 09:22:32 +08:00
Fabian Affolter
ed65ee50e0
Merge pull request #223537 from fabaff/zigpy-asynctest
python310Packages.zigpy: remove asynctest
2023-03-29 01:19:33 +02:00
Fabian Affolter
d6e84a3c50
Merge pull request #223533 from fabaff/bellows-asynctest
python310packages.bellows: remove asnyctest
2023-03-29 01:19:23 +02:00
Fabian Affolter
7eea1ba019
Merge pull request #223598 from r-ryantm/auto-update/python310Packages.google-cloud-os-config
python310Packages.google-cloud-os-config: 1.15.0 -> 1.15.1
2023-03-29 01:14:40 +02:00