2
0
mirror of https://github.com/NixOS/nixpkgs.git synced 2024-12-29 09:04:17 +00:00
nixpkgs/pkgs/servers/sql/postgresql/default.nix

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

382 lines
14 KiB
Nix
Raw Normal View History

2015-07-28 19:37:27 +00:00
let
generic =
# dependencies
2023-04-26 16:32:52 +00:00
{ stdenv, lib, fetchurl, makeWrapper, fetchpatch
, glibc, zlib, readline, openssl, icu, lz4, zstd, systemd, libossp_uuid
2023-04-26 16:32:52 +00:00
, pkg-config, libxml2, tzdata, libkrb5, substituteAll, darwin
# This is important to obtain a version of `libpq` that does not depend on systemd.
, enableSystemd ? lib.meta.availableOn stdenv.hostPlatform systemd && !stdenv.hostPlatform.isStatic
, gssSupport ? with stdenv.hostPlatform; !isWindows && !isStatic
# for postgresql.pkgs
, this, self, newScope, buildEnv
# source specification
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
, version, hash, psqlSchema
# for tests
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
, nixosTests, thisAttr
# JIT
, jitSupport ? false
, nukeReferences, patchelf, llvmPackages
, makeRustPlatform, buildPgxExtension, cargo, rustc
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
# detection of crypt fails when using llvm stdenv, so we add it manually
# for <13 (where it got removed: https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca)
, libxcrypt
}:
2018-12-08 05:32:26 +00:00
let
atLeast = lib.versionAtLeast version;
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
olderThan = lib.versionOlder version;
lz4Enabled = atLeast "14";
zstdEnabled = atLeast "15";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
stdenv' = if jitSupport then llvmPackages.stdenv else stdenv;
in stdenv'.mkDerivation rec {
pname = "postgresql";
inherit version;
2015-07-28 19:37:27 +00:00
src = fetchurl {
url = "mirror://postgresql/source/v${version}/${pname}-${version}.tar.bz2";
2022-08-11 21:36:01 +00:00
inherit hash;
2015-07-28 19:37:27 +00:00
};
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
hardeningEnable = lib.optionals (!stdenv'.cc.isClang) [ "pie" ];
2021-05-23 18:42:34 +00:00
Revert "postgresql: Add dev output" Firstly this creates a cycle in 9.6 .out <-> .dev after fixing the PGXS path. Secondly this breaks extension handling and the pg_config as it resolves a lot of paths relatively resulting in the following bogus output: BINDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/bin DOCDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc HTMLDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc INCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include PKGINCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include INCLUDEDIR-SERVER = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/locale MANDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/man SHAREDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 This can probably only be properly fixed by rewriting pg_config in large parts as it does not in any way respect splitting the postgres tree and assumes it can simply resolve paths relative to its location. This does not even handle symlinks: $ result-dev/bin/pg_config BINDIR = /home/robin/dev/nixpkgs-upstream/result-dev/bin DOCDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc HTMLDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc INCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include PKGINCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include INCLUDEDIR-SERVER = /home/robin/dev/nixpkgs-upstream/result-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/locale MANDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/man SHAREDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 cc @edolstra Revert "bacula: fix after postgresql output splitting" This reverts commit 13c9cbacdded00558084fda7037ca80849208e99. Revert "postgresql: fix pgxs dir in pg_config" This reverts commit 21998d597bd257c1a769e2be4b7a52b5b7e482a6. Revert "rubyGems.pg: fix postgresql path" This reverts commit e253ae7d3a4ab992d2432c87514e3333a3cf744d. Revert "kea: fix path to pg_config" This reverts commit 086c636eb7f894a55f752b6164dac4a87d8bcee9. Revert "php: fix build wrt. new postgres.dev build output" This reverts commit 2f23a83920107678ca1792a11d19e1cb37df3ace. Revert "gdal: fix path to pg_config" This reverts commit 032c50d45b3fd1ebb4af592b83e776ed7004b4c0. Revert "postgresql: Add dev output" This reverts commit b0280f598e4b3f6ebf33ad2115734e4735df443a.
2017-09-27 19:48:39 +00:00
outputs = [ "out" "lib" "doc" "man" ];
setOutputFlags = false; # $out retains configureFlags :-/
2015-07-28 19:37:27 +00:00
buildInputs = [
zlib
readline
openssl
libxml2
icu
]
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
++ lib.optionals (olderThan "13") [ libxcrypt ]
++ lib.optionals jitSupport [ llvmPackages.llvm ]
++ lib.optionals lz4Enabled [ lz4 ]
++ lib.optionals zstdEnabled [ zstd ]
++ lib.optionals enableSystemd [ systemd ]
++ lib.optionals gssSupport [ libkrb5 ]
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
++ lib.optionals (!stdenv'.isDarwin) [ libossp_uuid ];
2015-07-28 19:37:27 +00:00
nativeBuildInputs = [
makeWrapper
pkg-config
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
]
++ lib.optionals jitSupport [ llvmPackages.llvm.dev nukeReferences patchelf ];
2018-12-08 05:32:26 +00:00
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
enableParallelBuilding = !stdenv'.isDarwin;
2015-07-28 19:37:27 +00:00
separateDebugInfo = true;
buildFlags = [ "world" ];
2015-07-28 19:37:27 +00:00
env.NIX_CFLAGS_COMPILE = "-I${libxml2.dev}/include/libxml2";
# Otherwise it retains a reference to compiler and fails; see #44767. TODO: better.
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
preConfigure = "CC=${stdenv'.cc.targetPrefix}cc";
configureFlags = [
"--with-openssl"
"--with-libxml"
"--with-icu"
"--sysconfdir=/etc"
"--libdir=$(lib)/lib"
"--with-system-tzdata=${tzdata}/share/zoneinfo"
2020-11-10 11:43:09 +00:00
"--enable-debug"
(lib.optionalString enableSystemd "--with-systemd")
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
(if stdenv'.isDarwin then "--with-uuid=e2fs" else "--with-ossp-uuid")
] ++ lib.optionals lz4Enabled [ "--with-lz4" ]
++ lib.optionals zstdEnabled [ "--with-zstd" ]
2021-11-28 13:58:40 +00:00
++ lib.optionals gssSupport [ "--with-gssapi" ]
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
++ lib.optionals stdenv'.hostPlatform.isRiscV [ "--disable-spinlocks" ]
++ lib.optionals jitSupport [ "--with-llvm" ];
2015-07-28 19:37:27 +00:00
patches = [
2023-09-16 09:03:10 +00:00
(if atLeast "16" then ./patches/disable-normalize_exec_path.patch
else ./patches/disable-resolve_symlinks.patch)
./patches/less-is-more.patch
./patches/hardcode-pgxs-path.patch
./patches/specify_pkglibdir_at_runtime.patch
./patches/findstring.patch
2023-04-26 16:32:52 +00:00
(substituteAll {
src = ./locale-binary-path.patch;
locale = "${if stdenv.isDarwin then darwin.adv_cmds else lib.getBin stdenv.cc.libc}/bin/locale";
})
2023-08-17 19:33:38 +00:00
] ++ lib.optionals (stdenv'.hostPlatform.isMusl && atLeast "12") [
2023-04-26 16:32:52 +00:00
(fetchpatch {
url = "https://git.alpinelinux.org/aports/plain/main/postgresql14/icu-collations-hack.patch?id=56999e6d0265ceff5c5239f85fdd33e146f06cb7";
hash = "sha256-Yb6lMBDqeVP/BLMyIr5rmR6OkaVzo68cV/+cL2LOe/M=";
})
2023-08-17 19:33:38 +00:00
] ++ lib.optionals (stdenv'.hostPlatform.isMusl && atLeast "13") [
(if olderThan "14" then
fetchpatch {
url = "https://git.alpinelinux.org/aports/plain/main/postgresql13/disable-test-collate.icu.utf8.patch?id=69faa146ec9fff3b981511068f17f9e629d4688b";
hash = "sha256-IOOx7/laDYhTz1Q1r6H1FSZBsHCgD4lHvia+/os7CCo=";
}
else
fetchpatch {
url = "https://git.alpinelinux.org/aports/plain/main/postgresql14/disable-test-collate.icu.utf8.patch?id=56999e6d0265ceff5c5239f85fdd33e146f06cb7";
hash = "sha256-pnl+wM3/IUyq5iJzk+h278MDA9R0GQXQX8d4wJcB2z4=";
})
] ++ lib.optionals stdenv'.isLinux [
(if atLeast "13" then ./patches/socketdir-in-run-13.patch else ./patches/socketdir-in-run.patch)
];
2015-07-28 19:37:27 +00:00
installTargets = [ "install-world" ];
LC_ALL = "C";
postPatch = ''
# Hardcode the path to pgxs so pg_config returns the path in $out
substituteInPlace "src/common/config_info.c" --replace HARDCODED_PGXS_PATH "$out/lib"
'' + lib.optionalString jitSupport ''
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
# Force lookup of jit stuff in $out instead of $lib
substituteInPlace src/backend/jit/jit.c --replace pkglib_path \"$out/lib\"
substituteInPlace src/backend/jit/llvm/llvmjit.c --replace pkglib_path \"$out/lib\"
substituteInPlace src/backend/jit/llvm/llvmjit_inline.cpp --replace pkglib_path \"$out/lib\"
'';
2015-07-28 19:37:27 +00:00
postInstall =
''
Revert "postgresql: Add dev output" Firstly this creates a cycle in 9.6 .out <-> .dev after fixing the PGXS path. Secondly this breaks extension handling and the pg_config as it resolves a lot of paths relatively resulting in the following bogus output: BINDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/bin DOCDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc HTMLDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc INCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include PKGINCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include INCLUDEDIR-SERVER = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/locale MANDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/man SHAREDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 This can probably only be properly fixed by rewriting pg_config in large parts as it does not in any way respect splitting the postgres tree and assumes it can simply resolve paths relative to its location. This does not even handle symlinks: $ result-dev/bin/pg_config BINDIR = /home/robin/dev/nixpkgs-upstream/result-dev/bin DOCDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc HTMLDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc INCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include PKGINCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include INCLUDEDIR-SERVER = /home/robin/dev/nixpkgs-upstream/result-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/locale MANDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/man SHAREDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 cc @edolstra Revert "bacula: fix after postgresql output splitting" This reverts commit 13c9cbacdded00558084fda7037ca80849208e99. Revert "postgresql: fix pgxs dir in pg_config" This reverts commit 21998d597bd257c1a769e2be4b7a52b5b7e482a6. Revert "rubyGems.pg: fix postgresql path" This reverts commit e253ae7d3a4ab992d2432c87514e3333a3cf744d. Revert "kea: fix path to pg_config" This reverts commit 086c636eb7f894a55f752b6164dac4a87d8bcee9. Revert "php: fix build wrt. new postgres.dev build output" This reverts commit 2f23a83920107678ca1792a11d19e1cb37df3ace. Revert "gdal: fix path to pg_config" This reverts commit 032c50d45b3fd1ebb4af592b83e776ed7004b4c0. Revert "postgresql: Add dev output" This reverts commit b0280f598e4b3f6ebf33ad2115734e4735df443a.
2017-09-27 19:48:39 +00:00
moveToOutput "lib/pgxs" "$out" # looks strange, but not deleting it
moveToOutput "lib/libpgcommon*.a" "$out"
moveToOutput "lib/libpgport*.a" "$out"
moveToOutput "lib/libecpg*" "$out"
Revert "postgresql: Add dev output" Firstly this creates a cycle in 9.6 .out <-> .dev after fixing the PGXS path. Secondly this breaks extension handling and the pg_config as it resolves a lot of paths relatively resulting in the following bogus output: BINDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/bin DOCDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc HTMLDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc INCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include PKGINCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include INCLUDEDIR-SERVER = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/locale MANDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/man SHAREDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 This can probably only be properly fixed by rewriting pg_config in large parts as it does not in any way respect splitting the postgres tree and assumes it can simply resolve paths relative to its location. This does not even handle symlinks: $ result-dev/bin/pg_config BINDIR = /home/robin/dev/nixpkgs-upstream/result-dev/bin DOCDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc HTMLDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc INCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include PKGINCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include INCLUDEDIR-SERVER = /home/robin/dev/nixpkgs-upstream/result-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/locale MANDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/man SHAREDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 cc @edolstra Revert "bacula: fix after postgresql output splitting" This reverts commit 13c9cbacdded00558084fda7037ca80849208e99. Revert "postgresql: fix pgxs dir in pg_config" This reverts commit 21998d597bd257c1a769e2be4b7a52b5b7e482a6. Revert "rubyGems.pg: fix postgresql path" This reverts commit e253ae7d3a4ab992d2432c87514e3333a3cf744d. Revert "kea: fix path to pg_config" This reverts commit 086c636eb7f894a55f752b6164dac4a87d8bcee9. Revert "php: fix build wrt. new postgres.dev build output" This reverts commit 2f23a83920107678ca1792a11d19e1cb37df3ace. Revert "gdal: fix path to pg_config" This reverts commit 032c50d45b3fd1ebb4af592b83e776ed7004b4c0. Revert "postgresql: Add dev output" This reverts commit b0280f598e4b3f6ebf33ad2115734e4735df443a.
2017-09-27 19:48:39 +00:00
# Prevent a retained dependency on gcc-wrapper.
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
substituteInPlace "$out/lib/pgxs/src/Makefile.global" --replace ${stdenv'.cc}/bin/ld ld
Revert "postgresql: Add dev output" Firstly this creates a cycle in 9.6 .out <-> .dev after fixing the PGXS path. Secondly this breaks extension handling and the pg_config as it resolves a lot of paths relatively resulting in the following bogus output: BINDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/bin DOCDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc HTMLDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/doc INCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include PKGINCLUDEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include INCLUDEDIR-SERVER = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/locale MANDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share/man SHAREDIR = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 This can probably only be properly fixed by rewriting pg_config in large parts as it does not in any way respect splitting the postgres tree and assumes it can simply resolve paths relative to its location. This does not even handle symlinks: $ result-dev/bin/pg_config BINDIR = /home/robin/dev/nixpkgs-upstream/result-dev/bin DOCDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc HTMLDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/doc INCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include PKGINCLUDEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/include INCLUDEDIR-SERVER = /home/robin/dev/nixpkgs-upstream/result-dev/include/server LIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib PKGLIBDIR = /nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib LOCALEDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/locale MANDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share/man SHAREDIR = /home/robin/dev/nixpkgs-upstream/result-dev/share SYSCONFDIR = /etc/postgresql PGXS = /nix/store/csiq6zkn2c994kkdjln461jvgbka4yb1-postgresql-9.5.9-dev/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--prefix=/nix/store/irqqq4g173g9xrk1mh12kxv0s1d8dbyh-postgresql-9.5.9' '--with-openssl' '--with-libxml' '--sysconfdir=/etc' '--libdir=/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib' '--with-ossp-uuid' 'CC=gcc' CC = gcc CPPFLAGS = -D_GNU_SOURCE -I/nix/store/z6r0j2b4bcdfw3pck2x6ay0vvx0qzb92-libxml2-2.9.5-dev/include/libxml2 CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 CFLAGS_SL = -fPIC LDFLAGS = -L../../../src/common -L/nix/store/ighspl5sa3qi1zy7nkih0c9p73xjfqa6-libxml2-2.9.5/lib -Wl,--as-needed -Wl,-rpath,'/nix/store/85s9m7hpfzsl93wcb5gr6ii2km9mpx3z-postgresql-9.5.9-lib/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.5.9 cc @edolstra Revert "bacula: fix after postgresql output splitting" This reverts commit 13c9cbacdded00558084fda7037ca80849208e99. Revert "postgresql: fix pgxs dir in pg_config" This reverts commit 21998d597bd257c1a769e2be4b7a52b5b7e482a6. Revert "rubyGems.pg: fix postgresql path" This reverts commit e253ae7d3a4ab992d2432c87514e3333a3cf744d. Revert "kea: fix path to pg_config" This reverts commit 086c636eb7f894a55f752b6164dac4a87d8bcee9. Revert "php: fix build wrt. new postgres.dev build output" This reverts commit 2f23a83920107678ca1792a11d19e1cb37df3ace. Revert "gdal: fix path to pg_config" This reverts commit 032c50d45b3fd1ebb4af592b83e776ed7004b4c0. Revert "postgresql: Add dev output" This reverts commit b0280f598e4b3f6ebf33ad2115734e4735df443a.
2017-09-27 19:48:39 +00:00
2018-01-11 14:19:46 +00:00
if [ -z "''${dontDisableStatic:-}" ]; then
# Remove static libraries in case dynamic are available.
for i in $out/lib/*.a $lib/lib/*.a; do
2018-01-11 14:19:46 +00:00
name="$(basename "$i")"
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
ext="${stdenv'.hostPlatform.extensions.sharedLibrary}"
if [ -e "$lib/lib/''${name%.a}$ext" ] || [ -e "''${i%.a}$ext" ]; then
2018-01-11 14:19:46 +00:00
rm "$i"
fi
done
fi
'' + lib.optionalString jitSupport ''
# Move the bitcode and libllvmjit.so library out of $lib; otherwise, every client that
# depends on libpq.so will also have libLLVM.so in its closure too, bloating it
moveToOutput "lib/bitcode" "$out"
moveToOutput "lib/llvmjit*" "$out"
# In the case of JIT support, prevent a retained dependency on clang-wrapper
substituteInPlace "$out/lib/pgxs/src/Makefile.global" --replace ${self.llvmPackages.stdenv.cc}/bin/clang clang
nuke-refs $out/lib/llvmjit_types.bc $(find $out/lib/bitcode -type f)
# Stop out depending on the default output of llvm
substituteInPlace $out/lib/pgxs/src/Makefile.global \
--replace ${self.llvmPackages.llvm.out}/bin "" \
--replace '$(LLVM_BINPATH)/' ""
# Stop out depending on the -dev output of llvm
substituteInPlace $out/lib/pgxs/src/Makefile.global \
--replace ${self.llvmPackages.llvm.dev}/bin/llvm-config llvm-config \
--replace -I${self.llvmPackages.llvm.dev}/include ""
${lib.optionalString (!stdenv'.isDarwin) ''
# Stop lib depending on the -dev output of llvm
rpath=$(patchelf --print-rpath $out/lib/llvmjit.so)
nuke-refs -e $out $out/lib/llvmjit.so
# Restore the correct rpath
patchelf $out/lib/llvmjit.so --set-rpath "$rpath"
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
''}
2015-07-28 19:37:27 +00:00
'';
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
postFixup = lib.optionalString (!stdenv'.isDarwin && stdenv'.hostPlatform.libc == "glibc")
2016-11-22 21:48:18 +00:00
''
# initdb needs access to "locale" command from glibc.
wrapProgram $out/bin/initdb --prefix PATH ":" ${glibc.bin}/bin
'';
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
doCheck = !stdenv'.isDarwin;
2019-05-12 19:40:22 +00:00
# autodetection doesn't seem to able to find this, but it's there.
checkTarget = "check";
preCheck =
# On musl, comment skip the following tests, because they break due to
# ! ERROR: could not load library "/build/postgresql-11.5/tmp_install/nix/store/...-postgresql-11.5-lib/lib/libpqwalreceiver.so": Error loading shared library libpq.so.5: No such file or directory (needed by /build/postgresql-11.5/tmp_install/nix/store/...-postgresql-11.5-lib/lib/libpqwalreceiver.so)
# See also here:
# https://git.alpinelinux.org/aports/tree/main/postgresql/disable-broken-tests.patch?id=6d7d32c12e073a57a9e5946e55f4c1fbb68bd442
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
if stdenv'.hostPlatform.isMusl then ''
substituteInPlace src/test/regress/parallel_schedule \
--replace "subscription" "" \
--replace "object_address" ""
'' else null;
doInstallCheck = false; # needs a running daemon?
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
disallowedReferences = [ stdenv'.cc ];
2015-07-28 19:37:27 +00:00
passthru = let
jitToggle = this.override {
jitSupport = !jitSupport;
this = jitToggle;
};
in
{
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
inherit readline psqlSchema jitSupport;
withJIT = if jitSupport then this else jitToggle;
withoutJIT = if jitSupport then jitToggle else this;
pkgs = let
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
scope = {
postgresql = this;
stdenv = stdenv';
buildPgxExtension = buildPgxExtension.override {
stdenv = stdenv';
rustPlatform = makeRustPlatform {
stdenv = stdenv';
inherit rustc cargo;
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
};
};
};
newSelf = self // scope;
newSuper = { callPackage = newScope (scope // this.pkgs); };
in import ./packages.nix newSelf newSuper;
withPackages = postgresqlWithPackages {
inherit makeWrapper buildEnv;
postgresql = this;
}
this.pkgs;
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
tests = {
postgresql = nixosTests.postgresql-wal-receiver.${thisAttr};
} // lib.optionalAttrs jitSupport {
postgresql-jit = nixosTests.postgresql-jit.${thisAttr};
};
} // lib.optionalAttrs jitSupport {
inherit (llvmPackages) llvm;
2015-07-28 19:37:27 +00:00
};
meta = with lib; {
homepage = "https://www.postgresql.org";
2015-07-28 19:37:27 +00:00
description = "A powerful, open source object-relational database system";
license = licenses.postgresql;
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
maintainers = with maintainers; [ thoughtpolice danbst globin marsam ivan ma27 ];
platforms = platforms.unix;
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
# JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode
# (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for
# the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the
# host-platform.
#
# I managed to get a cross-build with JIT support working with
# `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the
# resulting LLVM IR isn't platform-independent this doesn't give you much.
# In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize
# a query, postgres would coredump with `Illegal instruction`.
broken = jitSupport && (stdenv.hostPlatform != stdenv.buildPlatform);
2015-07-28 19:37:27 +00:00
};
};
postgresqlWithPackages = { postgresql, makeWrapper, buildEnv }: pkgs: f: buildEnv {
name = "postgresql-and-plugins-${postgresql.version}";
paths = f pkgs ++ [
postgresql
postgresql.lib
postgresql.man # in case user installs this into environment
];
nativeBuildInputs = [ makeWrapper ];
2020-11-10 11:43:09 +00:00
# We include /bin to ensure the $out/bin directory is created, which is
# needed because we'll be removing the files from that directory in postBuild
# below. See #22653
pathsToLink = ["/" "/bin"];
# Note: the duplication of executables is about 4MB size.
# So a nicer solution was patching postgresql to allow setting the
2022-12-18 00:39:44 +00:00
# libdir explicitly.
postBuild = ''
mkdir -p $out/bin
rm $out/bin/{pg_config,postgres,pg_ctl}
cp --target-directory=$out/bin ${postgresql}/bin/{postgres,pg_config,pg_ctl}
wrapProgram $out/bin/postgres --set NIX_PGLIBDIR $out/lib
'';
passthru.version = postgresql.version;
passthru.psqlSchema = postgresql.psqlSchema;
};
2015-07-28 19:37:27 +00:00
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
mkPackages = self: {
# TODO: remove ahead of 23.11 branchoff
# "PostgreSQL 11 will stop receiving fixes on November 9, 2023"
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
postgresql_11 = self.callPackage generic {
version = "11.21";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
psqlSchema = "11.1"; # should be 11, but changing it is invasive
hash = "sha256-B7CDdHHV3XeyUWazRxjzuhCBa2rWHmkeb8VHzz/P+FA=";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
this = self.postgresql_11;
thisAttr = "postgresql_11";
inherit self;
};
2015-07-28 19:37:27 +00:00
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
postgresql_12 = self.callPackage generic {
version = "12.16";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
psqlSchema = "12";
hash = "sha256-xfH/96D5Ph7DdGQXsFlCkOzmF7SZXtlbjVJ68LoOOPM=";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
this = self.postgresql_12;
thisAttr = "postgresql_12";
inherit self;
};
2018-10-18 13:52:29 +00:00
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
postgresql_13 = self.callPackage generic {
version = "13.12";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
psqlSchema = "13";
hash = "sha256-DaHtzuNRS3vHum268MAEmeisFZBmjoeJxQJTpiSfIYs=";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
this = self.postgresql_13;
thisAttr = "postgresql_13";
inherit self;
};
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
postgresql_14 = self.callPackage generic {
version = "14.9";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
psqlSchema = "14";
hash = "sha256-sf47qbGn86ljfdFlbf2tKIkBYHP9TTXxO1AUPLu2qO8=";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
this = self.postgresql_14;
thisAttr = "postgresql_14";
inherit self;
};
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
postgresql_15 = self.callPackage generic {
version = "15.4";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
psqlSchema = "15";
hash = "sha256-uuxaS9xENzNmU7bLXZ7Ym+W9XAxYuU4L7O4KmZ5jyPk=";
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
this = self.postgresql_15;
thisAttr = "postgresql_15";
inherit self;
};
2023-09-16 09:03:10 +00:00
postgresql_16 = self.callPackage generic {
version = "16.0";
psqlSchema = "16";
hash = "sha256-356CPrIjMEROHUjlLMZRNaZSpv2zzjJePwhUkzn1G5k=";
this = self.postgresql_16;
thisAttr = "postgresql_16";
inherit self;
};
};
postgresql: implement opt-in JIT support Closes #150801 Note: I decided against resuming directly on #150801 because the conflict was too big (and resolving it seemed too error-prone to me). Also the `this`-refactoring could be done in an easier manner, i.e. by exposing JIT attributes with the correct configuration. More on that below. This patch creates variants of the `postgresql*`-packages with JIT[1] support. Please note that a lot of the work was derived from previous patches filed by other contributors, namely dasJ, andir and abbradar, hence the co-authored-by tags below. Effectively, the following things have changed: * For JIT variants an LLVM-backed stdenv with clang is now used as suggested by dasJ[2]. We need LLVM and CLang[3] anyways to build the JIT-part, so no need to mix this up with GCC's stdenv. Also, using the `dev`-output of LLVM and clang's stdenv for building (and adding llvm libs as build-inputs) seems more cross friendly to me (which will become useful when cross-building for JIT-variants will actually be supported). * Plugins inherit the build flags from the Makefiles in `$out/lib/pgxs/src` (e.g. `-Werror=unguarded-availability-new`). Since some of the flags are clang-specific (and stem from the use of the CLang stdenv) and don't work on gcc, the stdenv of `pkgs.postgresql` is passed to the plugins. I.e., plugins for non-JIT variants are built with a gcc stdenv on Linux and plugins for JIT variants with a clang stdenv. Since `plv8` hard-codes `gcc` as `$CC` in its Makefile[4], I marked it as broken for JIT-variants of postgresql only. * Added a test-matrix to confirm that JIT works fine on each `pkgs.postgresql_*_jit` (thanks Andi for the original test in #124804!). * For each postgresql version, a new attribute `postgresql_<version>_jit` (and a corresponding `postgresqlPackages<version>JitPackages`) are now exposed for better discoverability and prebuilt artifacts in the binary cache. * In #150801 the `this`-argument was replaced by an internal recursion. I decided against this approach because it'd blow up the diff even more which makes the readability way harder and also harder to revert this if necessary. Instead, it is made sure that `this` always points to the correct variant of `postgresql` and re-using that in an additional `.override {}`-expression is trivial because the JIT-variant is exposed in `all-packages.nix`. * I think the changes are sufficiently big to actually add myself as maintainer here. * Added `libxcrypt` to `buildInputs` for versions <v13. While building things with an LLVM stdenv, these versions complained that the extern `crypt()` symbol can't be found. Not sure what this is exactly about, but since we want to switch to libxcrypt for `crypt()` usage anyways[5] I decided to add it. For >=13 it's not relevant anymore anyways[6]. * JIT support doesn't work with cross-compilation. It is attempted to build LLVM-bytecode (`%.bc` is the corresponding `make(1)`-rule) for each sub-directory in `backend/` for the JIT apparently, but with a $(CLANG) that can produce binaries for the build, not the host-platform. I managed to get a cross-build with JIT support working with `depsBuildBuild = [ llvmPackages.clang ] ++ buildInputs`, but considering that the resulting LLVM IR isn't platform-independent this doesn't give you much. In fact, I tried to test the result in a VM-test, but as soon as JIT was used to optimize a query, postgres would coredump with `Illegal instruction`. A common concern of the original approach - with llvm as build input - was the massive increase of closure size. With the new approach of using the LLVM stdenv directly and patching out references to the clang drv in `$out` the effective closure size changes are: $ nix path-info -Sh $(nix-build -A postgresql_14) /nix/store/kssxxqycwa3c7kmwmykwxqvspxxa6r1w-postgresql-14.7 306.4M $ nix path-info -Sh $(nix-build -A postgresql_14_jit) /nix/store/xc7qmgqrn4h5yr4vmdwy56gs4bmja9ym-postgresql-14.7 689.2M Most of the increase in closure-size stems from the `lib`-output of LLVM $ nix path-info -Sh /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib /nix/store/5r97sbs5j6mw7qnbg8nhnq1gad9973ap-llvm-11.1.0-lib 349.8M which is why this shouldn't be enabled by default. While this is quite much because of LLVM, it's still a massive improvement over the simple approach of adding llvm/clang as build-inputs and building with `--with-llvm`: $ nix path-info -Sh $(nix-build -E ' with import ./. {}; postgresql.overrideAttrs ({ configureFlags ? [], buildInputs ? [], ... }: { configureFlags = configureFlags ++ [ "--with-llvm" ]; buildInputs = buildInputs ++ [ llvm clang ]; })' -j0) /nix/store/i3bd2r21c6c3428xb4gavjnplfqxn27p-postgresql-14.7 1.6G Co-authored-by: Andreas Rammhold <andreas@rammhold.de> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Nikolay Amiantov <ab@fmap.me> [1] https://www.postgresql.org/docs/current/jit-reason.html [2] https://github.com/NixOS/nixpkgs/pull/124804#issuecomment-864616931 & https://github.com/NixOS/nixpkgs/pull/150801#issuecomment-1467868321 [3] This fails with the following error otherwise: ``` configure: error: clang not found, but required when compiling --with-llvm, specify with CLANG= ``` [4] https://github.com/plv8/plv8/blob/v3.1.5/Makefile#L14 [5] https://github.com/NixOS/nixpkgs/pull/181764 [6] https://github.com/postgres/postgres/commit/c45643d618e35ec2fe91438df15abd4f3c0d85ca
2023-03-18 08:54:54 +00:00
in self:
let packages = mkPackages self; in
packages
// self.lib.mapAttrs'
(attrName: postgres: self.lib.nameValuePair "${attrName}_jit" (postgres.override rec {
jitSupport = true;
thisAttr = "${attrName}_jit";
this = self.${thisAttr};
}))
packages