Commit Graph

963 Commits

Author SHA1 Message Date
Greg Roodt
42d28c2a77 python: 3.9.0 -> 3.9.1 2020-12-12 19:32:08 +01:00
Frederik Rietdijk
76966f8ddd pythonPackages.pip-install-hook: remove --build option
Pip had the option --build to build in a custom or temporary directory.
Nowadays, pip just listens to TMPDIR, which we already set.

This option was deprecated and is removed in pip 20.3.
2020-12-01 14:44:26 +01:00
Frederik Rietdijk
9a63b3d3d6
Merge pull request #104781 from NixOS/staging-next
Staging next
2020-11-30 18:27:29 +01:00
adisbladis
49b8738a5f
python: Propagate packageOverrides to pythonForBuild
When overriding versions of build tools injected via hooks
`packageOverrides` was not taken into account and 2 incompatible
versions of the same package (`wheel` in this case) ended up in the
closure, causing the builds to fail.
2020-11-30 11:12:59 +01:00
Frederik Rietdijk
0d8491cb2b Merge master into staging-next 2020-11-29 13:51:10 +01:00
Frederik Rietdijk
455774e546 Python: add todo note regarding pythonForBuild 2020-11-28 17:36:23 +01:00
Frederik Rietdijk
5804c8a7f5 Python splicing: splice package sets, not Python derivation
In the original commit the various builds of Python were added to
`otherSplices`, instead of the intended Python package sets.
2020-11-28 17:36:23 +01:00
Frederik Rietdijk
6cf25f9dbd Python: rename parameters and arguments passed to passthru
As part of the splicing the build/host/target combinations of the interpreter
need to be passed around internally. The chosen names were not very clear,
implying they were package sets whereas actually there were derivations.
2020-11-28 17:36:23 +01:00
Frederik Rietdijk
8220b0449c python.tests: use self.callPackage instead of super.callPackage
super was incorrectly possible until https://github.com/NixOS/nixpkgs/pull/104201
got merged.
2020-11-28 16:57:07 +01:00
Frederik Rietdijk
cce2fd547b Python: use pythonPackagesBuildHost instead of pythonForBuild
Follow-up to #104201, related to #105113.
2020-11-28 16:36:03 +01:00
Jonathan Ringer
0a4575a1da python/hooks/pythonNamespaces: fix __pycache__ being empty, or not existing 2020-11-26 13:51:05 -08:00
Frederik Rietdijk
0a12b8a5e9 Revert "Revert "Revert "cpython: fix finding headers when cross-compiling extension modules"""
This still does not function without issues. E.g., bootstrapped-pip fails with Python 3.6 and 3.7
as well as 3.8 on 32-bit.

Because this is a stdenv-rebuild it needs to be tested significantly better

This reverts commit 6100bc29f7.
2020-11-25 10:34:23 +01:00
Jonathan Ringer
f5d9dd3050 python/hooks/pythonNamespaces: fix __pycache__ logic 2020-11-23 08:33:04 -08:00
Frederik Rietdijk
95d9ff16f3 python.tests: not when cross-compiling
Running the tests is not possible when cross-compiling.
2020-11-23 15:29:23 +01:00
Frederik Rietdijk
6100bc29f7 Revert "Revert "cpython: fix finding headers when cross-compiling extension modules""
Moving the sysconfig file caused spidermonkey_78 to fail to build. We now symlink it instead.

This reverts commit 6a23dde75b.
2020-11-23 15:29:23 +01:00
Frederik Rietdijk
7726f81b13 python.tests: use self.callPackage instead of super.callPackage
super was incorrectly possible until https://github.com/NixOS/nixpkgs/pull/104201
got merged.
2020-11-22 13:17:24 +01:00
Frederik Rietdijk
6a23dde75b Revert "cpython: fix finding headers when cross-compiling extension modules"
This breaks virtualenv https://github.com/NixOS/nixpkgs/issues/104483.
We should probably not move `_sysconfigdata` but just copy it for cross to
another place or output.

This reverts commit 11806b6ede.
2020-11-22 13:13:23 +01:00
Jonathan Ringer
8d210e2ea4 python3.pkgs.pipInstallHook: don't warn on script installation 2020-11-20 12:25:52 -08:00
Frederik Rietdijk
27d3a18ac7 buildPython*: don't catch conflicts when cross-compiling
Hack until we fix PYTHONPATH.
2020-11-19 20:59:16 +01:00
Ben Wolsieffer
11806b6ede cpython: fix finding headers when cross-compiling extension modules 2020-11-19 20:42:58 +01:00
Frederik Rietdijk
ea7b8978ef Merge master into staging-next 2020-11-19 20:08:15 +01:00
John Ericson
b57c5d4456 python: Use makeScopeWithSplicing
Now non-`buildInputs` that are python packages should be resolved
correctly.
2020-11-19 11:58:07 -05:00
John Ericson
330d468002 python: Use packages from previous stage in setup-hooks
This makes sense as these are tools we want to run at build time.
2020-11-19 11:58:03 -05:00
Frederik Rietdijk
da12fc6838 Merge staging-next into staging 2020-11-18 15:36:56 +01:00
Bernardo Meurer
7edf1ced82 pypy: enable aarch64-linux platform 2020-11-18 09:55:33 +01:00
Anders Kaseorg
9e09ad5182 pypy, pypy3: 7.3.1 → 7.3.2
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-11-18 00:21:41 -08:00
Anders Kaseorg
b7836e5ca6 pypy, pypy3: Follow Heptapod move for download URL
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-11-18 00:21:41 -08:00
Frederik Rietdijk
9fc4a4db5a Merge staging-next into staging 2020-11-13 20:59:08 +01:00
Frederik Rietdijk
3fceafce1e buildPython*: remove pythonRecompileBytecodeHook as dependency
This hook was added to get reproducible bytecode. Because it was causing
issues it was disabled, but still kept as a dependency. Now the main
issue with bytecode reproducibility has been resolved by updating pip to
20.2.4, we remove this hook as a dependency.

If a package with Python code is not yet reproducible, one could add
this hook to `nativeBuildInputs`.
2020-11-12 19:32:30 +01:00
Jonathan Ringer
5a5122418a python310: 3.10.0a1 -> 3.10.0a2 2020-11-11 23:13:44 -08:00
Frederik Rietdijk
20f001c01e Merge master into staging-next 2020-11-09 14:33:52 +01:00
Samuel Gräfenstein
22b73d1282 pkgs/*: add final newline to .nix files 2020-11-09 11:39:28 +00:00
Frederik Rietdijk
61a04f735a python3.pkgs.pip: 20.1.1 -> 20.2.4
Reproducible builds of pyproject projects using pip is resolved.

Fixes https://github.com/pypa/pip/issues/7808
Fixes https://github.com/NixOS/nixpkgs/issues/81441

The more recent c409f69480 caused trouble
with pyproject troubles and had to be reverted anyway.
https://github.com/NixOS/nixpkgs/pull/102222#issuecomment-722380794

Revert "pythonPackages.pip: make reproducible (#102222)"

This reverts commit c409f69480.

Revert "python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR"

This reverts commit aedbade43e.
2020-11-05 15:07:21 +01:00
Frederik Rietdijk
6ddd6b4914 Revert "Merge pull request #93083 from risicle/ris-cpython-debug"
This increase the closure size by 30 MB.

https://github.com/NixOS/nixpkgs/pull/93083#issuecomment-718025366

This reverts commit 4b340cbbb4, reversing
changes made to 47e499f7a4.
2020-11-03 12:50:05 +01:00
zimbatm
c409f69480
pythonPackages.pip: make reproducible (#102222)
The previous attempt wasn't covering all of the bases. It relied on
invoking that pip-install-hook, and didn't apply to pip itself.

The core issue is that the generated .pyc files embed some of the
temporary paths, which are randomly generated. See
https://r13y.com/diff/bf8c3ca3148ebff9ecf41f294cc60b9f209c006d49699e356969ff32d736f1c6-8806a7cca91fdd300e48736bfcd57c4d0b54c1cc2fd61609f35143170862b59c.html

In this new attempt, the approach is to patch the TempFile
implementation directly, so that it creates stable temporary
directories. We also assume that if SOURCE_DATE_EPOCH is set, we are in
a scenario where reproducible builds are desirable and enter that
branch.

See also https://github.com/pypa/pip/issues/7808
2020-10-31 21:24:36 +01:00
Jonathan Ringer
67d2de5ffa python310: init at 3.10a1 2020-10-19 09:31:27 -07:00
Jonathan Ringer
9c9418151d update-python-libraries: allow for top-level attrs to be updated with github fetcher 2020-10-15 12:10:08 -07:00
Frederik Rietdijk
9e1943edc0 Merge master into staging-next 2020-10-13 19:34:34 +02:00
Jonathan Ringer
2a9c3ec083
update-python-libraries: make script updateScript friendly 2020-10-11 12:03:18 -07:00
Frederik Rietdijk
692d219a93 Merge staging-next into staging 2020-10-06 10:25:58 +02:00
Jonathan Ringer
4eaf246f24 update-python-libraries: add github fetcher 2020-10-06 00:55:43 -07:00
Raphael Borun Das Gupta
3696bc0e89 python39: 3.9.0rc2 -> 3.9.0 2020-10-06 08:32:37 +02:00
Jan Tojnar
32b4375f10
Merge branch 'staging-next' into staging 2020-09-29 00:12:29 +02:00
Frederik Rietdijk
7c411b3729 python38: 3.8.5 -> 3.8.6 2020-09-27 13:07:49 +02:00
Frederik Rietdijk
74f3e9cce1 python39: 3.9.0rc1 -> 3.9.0rc2 2020-09-27 11:48:00 +02:00
Johannes Rosenberger
7abb57c7b5 fix passing qt5 version to pythonInterpreters
fixes c88f3adb17, which resulted in
qt 5.15 being used in pythonPackages, despite 5.14 being
declared, and adapts qutebrowser accordingly.

'callPackage { pkgs = pkgs // { … }; }' does not work, because
it does not take into account the recursive evaluation of nixpkgs:

`pkgs/development/interpreters/python/default.nix` calls
`pkgs/top-level/python-packages.nix` with `callPackage`.
Thus, even if the former gets passed the updated `pkgs`,
the latter always gets passed `pkgs.pkgs`.

For the change in the qt5 version to apply consistently, 'pkgs.extend'
must be used.

qutebrowser only used the right qt5 version (5.15) because all
pythonPackages used it anyway.
2020-09-22 21:03:43 +02:00
Drew Risinger
60df61b67e mk-python-derivation: document pyproject format
Optional setting of format == "pyproject", "egg" had been documented
in the manual, but they weren't listed in the function header for
``mk-python-derivation.nix``.
2020-09-16 17:38:41 +02:00
Frederik Rietdijk
af81d39b87 Merge staging-next into staging 2020-09-04 20:03:30 +02:00
Jonathan Ringer
762434fb02 python39: 3.9.0b5 -> 3.9.0rc1 2020-09-03 23:36:28 -07:00
Frederik Rietdijk
e29c1e42e0 Merge staging-next into staging 2020-08-31 19:46:33 +02:00
Frederik Rietdijk
5d8dd5c259 python.tests: don't mix the two types of tests
The integration tests code was included in the environment tests. After
this commit it is hopefully clearer what belongs together.
2020-08-30 12:59:24 +02:00
Frederik Rietdijk
2248a7422f python36: 3.6.11 -> 3.6.12 2020-08-30 08:59:42 +02:00
Spencer Baugh
780caf6690 python37: 3.7.8 -> 3.7.9
Release notes:
https://docs.python.org/release/3.7.9/whatsnew/changelog.html

Most notable is the fixed behavior of asynchronous generators; without
this fix, they fail to work in some scenarios.
2020-08-30 08:52:06 +02:00
Jan Tojnar
91104b5417
Merge branch 'master' into staging-next 2020-08-23 02:00:50 +02:00
Lassulus
7d03cf2c8d
Merge pull request #91667 from DavHau/fix-indentation
mk-python-derivation.nix: fix indentation
2020-08-21 17:07:44 +02:00
Frederik Rietdijk
0ac85bc455 Merge master into staging-next 2020-08-17 14:54:39 +02:00
Alvar Penning
b7ce309e6c pythonFull: add BlueZ support
This commit introduces two changes.

First, cpython gets optional BlueZ support, which is needed for
AF_BLUETOOTH sockets. Therefore bluezSupport was added as a parameter.

Second, the call to the pythonFull packages has been adjusted. The
Python packages have a self-reference called self. This was not adjusted
for the override. As a result, Python packages for this special version
of Python were not built with the overridden Python, but with the
original one.
2020-08-15 18:08:20 +02:00
Frederik Rietdijk
cfe6081cee Merge staging-next into staging 2020-08-15 09:12:42 +02:00
Frederik Rietdijk
4b340cbbb4
Merge pull request #93083 from risicle/ris-cpython-debug
cpython: add separateDebugInfo, allow use of gdb libpython
2020-08-15 08:52:42 +02:00
Daniël de Kok
900b2d5a6c python3: fix impure /bin/sh call in subprocess 2020-08-15 08:34:02 +02:00
Jonathan Ringer
91bec61635
python35: remove 2020-08-12 21:57:18 -07:00
Jan Tojnar
11da469fa5
Merge branch 'staging-next' into staging 2020-08-11 16:18:42 +02:00
Christian Kauhaus
a14859c686 python: Apply patch for CVE-2019-20907
Incluing the patch file in-tree because the upstream patch is not
intended to apply for Python 2.

Re #94004
2020-08-11 16:05:43 +02:00
Anders Kaseorg
b31e4a20a4 pypy, pypy3: 7.1.1 -> 7.3.1
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-10 15:42:42 -07:00
Anders Kaseorg
6f2ec6d967 pythonInterpreters.pypy{27,36}_prebuilt: 7.1.1 -> 7.3.1
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-10 15:42:42 -07:00
Matthew Bauer
d0677e6d45 treewide: add warning comment to “boot” packages
This adds a warning to the top of each “boot” package that reads:

  Note: this package is used for bootstrapping fetchurl, and thus cannot
  use fetchpatch! All mutable patches (generated by GitHub or cgit) that
  are needed here should be included directly in Nixpkgs as files.

This makes it clear to maintainer that they may need to treat this
package a little differently than others. Importantly, we can’t use
fetchpatch here due to using <nix/fetchurl.nix>. To avoid having stale
hashes, we need to include patches that are subject to changing
overtime (for instance, gitweb’s patches contain a version number at
the bottom).
2020-07-31 08:56:53 +02:00
Mario Rodas
ced8ec8488 python39: 3.9.0a4 -> 3.9.0b5 2020-07-26 18:45:54 +02:00
Mario Rodas
79590e27d4 python38: 3.8.3 -> 3.8.5 2020-07-26 18:45:54 +02:00
Robert Scott
ede2e00c9f cpython: expose gdb libpython.py
used together with cpython's debugging symbols, this allows inspection of
the python stack of cpython programs in gdb. this file is a little
different from the rest of the python output by this package, in that it's
not intended to be run by the current python being built, instead by the
python being used by the gdb in question, which could be very different.
therefore placed in its own, but hopefully logical & predictable location.
2020-07-15 19:39:19 +01:00
Robert Scott
6d19ab339a cpython: set separateDebugInfo 2020-07-13 22:30:55 +01:00
Frederik Rietdijk
7935bf793e python37: 3.7.7 -> 3.7.8 2020-07-04 18:13:44 +02:00
Frederik Rietdijk
60fb772ae2 python36: 3.6.10 -> 3.6.11 2020-07-04 18:13:44 +02:00
misuzu
aedbade43e python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR 2020-07-02 17:27:13 +02:00
DavHau
d1c1a0c656 fix indentation in mk-python-derivation.nix 2020-06-27 14:26:03 +00:00
Frederik Rietdijk
f03734efa0 python3: improve cross-compilation of extension modules, fixes #91171 2020-06-21 16:29:57 +02:00
Frederik Rietdijk
b7aec77a7b buildPythonPackage: don't recompile bytecode
There are too many regressions. Instead of reverting all the work that has been
done on this so far, let's just disable it Python-wide. That way we can
investigate and fix it easier.
2020-06-17 14:54:03 +02:00
Jonathan Ringer
b6654d9d6c python2Packages: default to dontUsePythonRecompileBytecode = true; 2020-06-16 13:02:50 -07:00
Frederik Rietdijk
2e4b4e3300 Merge staging-next into staging 2020-06-13 11:03:26 +02:00
Frederik Rietdijk
818cf7827b buildPythonPackage: recompile bytecode for reproducibility
Due to a change in pip the unpacked wheels are no longer reproducible.
We recompile the bytecode to cleanup this error.
https://github.com/NixOS/nixpkgs/issues/81441
2020-06-13 10:36:28 +02:00
Frederik Rietdijk
a7ea828f09 pythonRemoveBinBytecodeHook: fix explanation 2020-06-13 10:20:14 +02:00
Frederik Rietdijk
4087d3fe41 python: don't use optimizations on Darwin
Also, don't use autoreconfHook on Darwin with Python 3.
Darwin builds are still impure and fail with

    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: object file (/nix/store/0lsij4jl35bnhqhdzla8md6xiswgig5q-Libsystem-osx-10.12.6/lib/crt1.10.6.o) was built for newer OSX version (10.12) than being linked (10.6)
    DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
    if test $? -ne 0 ; then \
            echo "generate-posix-vars failed" ; \
            rm -f ./pybuilddir.txt ; \
            exit 1 ; \
    fi
    /nix/store/dsb7d4dwxk6bzlm845z2zx6wp9a8bqc1-bash-4.4-p23/bin/bash: line 5: 72015 Killed: 9               DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars
    generate-posix-vars failed
    make: *** [Makefile:592: pybuilddir.txt] Error 1
2020-06-12 18:29:08 +02:00
Daniël de Kok
1e2b6695cf pythonPackages.setuptoolsBuildHook: do not build in an isolated environment
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:

https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support

For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.

This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
2020-06-06 10:05:26 +02:00
Daniël de Kok
e2309df85e pythonPackages.pipBuildHook: do not build in an isolated environment
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:

https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support

For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.

This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
2020-06-06 10:05:26 +02:00
Frederik Rietdijk
1c68570ab2 Merge staging-next into staging 2020-06-05 19:42:16 +02:00
Frederik Rietdijk
43f71029cc Merge master into staging-next 2020-06-05 19:40:53 +02:00
Frederik Rietdijk
913bee36ed python3Minimal: override python38, not python3
This avoids an infinite recursion, accidentally introduced in b7ff746540.
2020-06-05 16:46:40 +02:00
Frederik Rietdijk
a337c44db6 python3Minimal: disable optimizations
No point for the bootstrapping.
2020-06-04 20:53:31 +02:00
Frederik Rietdijk
bcf03e8cd2 Revert "cpython: Optimize dynamic symbol tables, for a 6% speedup."
ofborg does not like fetching patches when the derivation is used during bootstrapping.

This reverts commit 480c8d1991.
2020-06-04 20:36:31 +02:00
Frederik Rietdijk
a2be64bf13
Merge pull request #84072 from gnprice/python-build
cpython: Use optimizations, for a 25% speedup.
2020-06-04 18:31:07 +02:00
Frederik Rietdijk
b7ff746540 python3: now points to python38
Note this also means python3Minimal is now also Python 3.8.

This reverts commit eb1369670b and adds more.
2020-06-04 18:08:29 +02:00
Luflosi
2379e36124 python39: fix build on macOS
Basically the same changes as in 81d15948cc but for python3.9 instead of python3.8.
2020-06-04 17:11:29 +02:00
Frederik Rietdijk
0367fa630d python38: 3.8.2 -> 3.8.3 2020-05-27 12:10:25 +02:00
Frederik Rietdijk
f17001afd8 Python: fix virtualenv with Python 2 2020-05-24 10:43:24 +02:00
Frederik Rietdijk
98bcf5d8da Python tests: fix use of is_virtualenv
Too many tests set it.
2020-05-24 10:43:24 +02:00
Frederik Rietdijk
c778596f56 Merge master into staging-next 2020-05-24 10:03:22 +02:00
Frederik Rietdijk
2de446e0b8 python.tests: also test virtualenv
Test whether creating a virtualenv functions.
2020-05-23 18:15:45 +02:00
adisbladis
203f382a4a
pypy: Remove bootstrap python from closure 2020-05-23 11:47:11 +01:00
Jan Tojnar
7f40cfd97b
Merge branch 'master' into staging-next 2020-05-18 21:09:27 +02:00
Jon
15b3d9d277
python3Packages.venvShellHook: add postVenvCreation (#87850)
* python3Packages.venvShellHook: add postVenvCreation

* python: docs: add postVenvCreation explaination
2020-05-16 09:34:11 +02:00
Greg Price
480c8d1991 cpython: Optimize dynamic symbol tables, for a 6% speedup.
I took a close look at how Debian builds the Python interpreter,
because I noticed it ran substantially faster than the one in nixpkgs
and I was curious why.

One thing that I found made a material difference in performance was
this pair of linker flags (passed to the compiler):

    -Wl,-O1 -Wl,-Bsymbolic-functions

In other words, effectively the linker gets passed the flags:

    -O1 -Bsymbolic-functions

Doing the same thing in nixpkgs turns out to make the interpreter
run about 6% faster, which is quite a big win for such an easy
change.  So, let's apply it.

---

I had not known there was a `-O1` flag for the *linker*!
But indeed there is.

These flags are unrelated to "link-time optimization" (LTO), despite
the latter's name.  LTO means doing classic compiler optimizations
on the actual code, at the linking step when it becomes possible to
do them with cross-object-file information.  These two flags, by
contrast, cause the linker to make certain optimizations within the
scope of its job as the linker.

Documentation is here, though sparse:
  https://sourceware.org/binutils/docs-2.31/ld/Options.html

The meaning of -O1 was explained in more detail in this LWN article:
  https://lwn.net/Articles/192624/
Apparently it makes the resulting symbol table use a bigger hash
table, so the load factor is smaller and lookups are faster.  Cool.

As for -Bsymbolic-functions, the documentation indicates that it's a
way of saving lookups through the symbol table entirely.  There can
apparently be situations where it changes the behavior of a program,
specifically if the program relies on linker tricks to provide
customization features:
  https://bugs.launchpad.net/ubuntu/+source/xfe/+bug/644645
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637184#35
But I'm pretty sure CPython doesn't permit that kind of trick: you
don't load a shared object that tries to redefine some symbol found
in the interpreter core.

The stronger reason I'm confident using -Bsymbolic-functions is
safe, though, is empirical.  Both Debian and Ubuntu have been
shipping a Python built this way since forever -- it was introduced
for the Python 2.4 and 2.5 in Ubuntu "hardy", and Debian "lenny",
released in 2008 and 2009.  In those 12 years they haven't seen a
need to drop this flag; and I've been unable to locate any reports
of trouble related to it, either on the Web in general or on the
Debian bug tracker.  (There are reports of a handful of other
programs breaking with it, but not Python/CPython.)  So that seems
like about as thorough testing as one could hope for.

---

As for the performance impact: I ran CPython upstream's preferred
benchmark suite, "pyperformance", in the same way as described in
the previous commit.  On top of that commit's change, the results
across the 60 benchmarks in the suite are:

The median is 6% faster.

The middle half (aka interquartile range) is from 4% to 8% faster.

Out of 60 benchmarks, 3 come out slower, by 1-4%.  At the other end,
5 are at least 10% faster, and one is 17% faster.

So, that's quite a material speedup!  I don't know how big the
effect of these flags is for other software; but certainly CPython
tends to do plenty of dynamic linking, as that's how it loads
extension modules, which are ubiquitous in the stdlib as well as
popular third-party libraries.  So perhaps that helps explain why
optimizing the dynamic linker has such an impact.
2020-05-13 21:24:30 -07:00
Greg Price
52c04b0347 cpython: Use autoreconfHook to rebuild configure script.
In particular this will let us use patches that apply to configure.ac.
2020-05-13 21:23:48 -07:00
Greg Price
f8a8243bd3 cpython: Use --enable-optimizations, for a 16% speedup.
Without this flag, the configure script prints a warning at the end,
like this (reformatted):

  If you want a release build with all stable optimizations active
  (PGO, etc), please run ./configure --enable-optimizations

We're doing a build to distribute to people for day-to-day use,
doing things other than developing the Python interpreter.  So
that's certainly a release build -- we're the target audience for
this recommendation.

---

And, trying it out, upstream isn't kidding!  I ran the standard
benchmark suite that the CPython developers use for performance
work, "pyperformance".  Following its usage instructions:
  https://pyperformance.readthedocs.io/usage.html
I ran the whole suite, like so:

  $ nix-shell -p ./result."$variant" --run '
      cd $(mktemp -d); python -m venv venv; . venv/bin/activate
      pip install pyperformance
      pyperformance run -o ~/tmp/result.'"$variant"'.json
    '

and then examined the results with commands like:

  $ python -m pyperf compare_to --table -G \
      ~/tmp/result.{$before,$after}.json

Across all the benchmarks in the suite, the median speedup was 16%.
(Meaning 1.16x faster; 14% less time).

The middle half of them ranged from a 13% to a 22% speedup.

Each of the 60 benchmarks in the suite got faster, by speedups
ranging from 3% to 53%.

---

One reason this isn't just the default to begin with is that, until
recently, it made the build a lot slower.  What it does is turn on
profile-guided optimization, which means first build for profiling,
then run some task to get a profile, then build again using the
profile.  And, short of further customization, the task it would use
would be nearly the full test suite, which includes a lot of
expensive and slow tests, and can easily take half an hour to run.

Happily, in 2019 an upstream developer did the work to carefully
select a more appropriate set of tests to use for the profile:
  https://github.com/python/cpython/commit/4e16a4a31
  https://bugs.python.org/issue36044
This suite takes just 2 minutes to run.  And the resulting final
build is actually slightly faster than with the much longer suite,
at least as measured by those standard "pyperformance" benchmarks.
That work went into the 3.8 release, but the same list works great
if used on older releases too.

So, start passing that --enable-optimizations flag; and backport
that good-for-PGO set of tests, so that we use it on all releases.
2020-05-11 23:37:04 -07:00
Jonathan Ringer
884436b254 pythonPackages.pytestCheckHook: disable setuptoolsCheckPhase 2020-05-11 22:12:08 +02:00
Josef Kemetmüller
3818cd9049 python: Fix creating RPMs from Python packages
This should enable (manual) building of RPMs from python projects using
the `python setup.py bdist_rpm` command on systems where `rpmbuild` is
not located in `/usr/bin/`. (e.g. NixOS)
The discovery of the rpmbuild command was fixed upstream in Python 3.8,
so this commit backports the relevant patch to our currently supported
Python 3 versions.

Fixes: #85204
2020-05-09 11:15:28 +02:00
Pavol Rusnak
420124adf8 python: remove isPy33, isPy34 2020-05-04 18:49:45 -07:00
Frederik Rietdijk
6f873e98f4 Python integration tests: disable for older python 3 versions
because the package that is used as part of the test does not support
older versions.
2020-04-25 07:59:37 +02:00
Frederik Rietdijk
71171b3225 Python tests: test venv from a nix env with Python 3.8
This test was disabled because it did not function yet, however,
apparently it does with 3.8.
2020-04-25 07:59:37 +02:00
Frederik Rietdijk
cf1a68360e python2: 2.7.17 -> 2.7.18 2020-04-21 11:21:39 +02:00
Michael Reilly
84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
Jan Tojnar
3e0f4e202f
Merge branch 'master' into staging-next 2020-03-31 21:32:15 +02:00
Greg Price
9d8831c8fe cpython: Drop unrecognized --with-threads configure flag.
The ./configure script prints a warning when passed this flag,
starting with 3.7:

  configure: WARNING: unrecognized options: --with-threads

The reason is that there's no longer such a thing as a build
without threads.

Eliminate the warning, by only passing the flag on the older releases
that accept it.

Upstream change and discussion:
  https://github.com/python/cpython/commit/a6a4dc816
  https://bugs.python.org/issue31370
2020-03-30 22:56:48 -07:00
Frederik Rietdijk
c392d70518 pkgsStatic.python3: fix build 2020-03-30 17:06:38 +02:00
Frederik Rietdijk
a36be028f5 Merge staging-next into staging 2020-03-28 21:15:15 +01:00
Jonathan Ringer
279438e7f8 python: add use-pkgs-prefix option to update script 2020-03-27 01:47:30 -07:00
Jonathan Ringer
a9c994ad0b python: add pythonNamespacesHook 2020-03-18 12:20:51 -07:00
Frederik Rietdijk
7447fff95a
Fix sys.prefix in case of a Nix env
The prefix will now be correct in case of Nix env.

Note, however, that creating a venv from a Nix env still does not function. This does not seem to be possible
with the current approach either, because venv will copy or symlink our Python wrapper. In case it symlinks
(the default) it won't see a pyvenv.cfg. If it is copied I think it should function but it does not...
2020-03-14 21:39:32 +00:00
adisbladis
753122388d
Python: Add integration test verifying NIX_PYTHONPATH with Mypy 2020-03-14 21:39:32 +00:00
adisbladis
05571d3059
Python Add test for NIX_PYTHONPREFIX 2020-03-14 21:39:31 +00:00
adisbladis
d88a7735d2
Python: introduce NIX_PYTHONPREFIX in order to set site.PREFIXES
This is needed in case of `python.buildEnv` to make sure site.PREFIXES
does not only point to the unwrapped executable prefix.

--------------------------------------------------------------------------------

This PR is a story where your valiant hero sets out on a very simple adventure but ends up having to slay dragons, starts questioning his own sanity and finally manages to gain enough knowledge to slay the evil dragon and finally win the proverbial price.

It all started out on sunny spring day with trying to tackle the Nixops plugin infrastructure and make that nice enough to work with.

Our story begins in the shanty town of [NixOps-AWS](https://github.com/nixos/nixops-aws) where [mypy](http://mypy-lang.org/) type checking has not yet been seen.

As our deuteragonist (@grahamc) has made great strides in the capital city of [NixOps](https://github.com/nixos/nixops) our hero wanted to bring this out into the land and let the people rejoice in reliability and a wonderful development experience.

The plugin work itself was straight forward and our hero quickly slayed the first small dragon, at this point things felt good and our hero thought he was going to reach the town of NixOps-AWS very quickly.

But alas! Mypy did not want to go, it said:
`Cannot find implementation or library stub for module named 'nixops'`

Our hero felt a small sliver of life escape from his body. Things were not going to be so easy.

After some frustration our hero discovered there was a [rule of the land of Python](https://www.python.org/dev/peps/pep-0561/) that governed the import of types into the kingdom, more specificaly a very special document (file) called `py.typed`.
Things were looking good.

But no, what the law said did not seem to match reality. How could things be so?

After some frustrating debugging our valiant hero thought to himself "Hmm, I wonder if this is simply a Nix idiosyncrasy", and it turns out indeed it was.
Things that were working in the blessed way of the land of Python (inside a `virtualenv`) were not working the way they were from his home town of Nix (`nix-shell` + `python.withPackages`).

After even more frustrating attempts at reading the mypy documentation and trying to understand how things were supposed to work our hero started questioning his sanity.
This is where things started to get truly interesting.

Our hero started to use a number of powerful weapons, both forged in the land of Python (pdb) & by the mages of UNIX (printf-style-debugging & strace).

After first trying to slay the dragon simply by `strace` and a keen eye our hero did not spot any weak points.
Time to break out a more powerful sword (`pdb`) which also did not divulge any secrets about what was wrong.

Our hero went back to the `strace` output and after a fair bit of thought and analysis a pattern started to emerge. Mypy was looking in the wrong place (i.e. not in in the environment created by `python.withPackages` but in the interpreter store path) and our princess was in another castle!

Our hero went to the pub full of old grumpy men giving out the inner workings of the open source universe (Github) and acquired a copy of Mypy.
He littered the code with print statements & break points.
After a fierce battle full of blood, sweat & tears he ended up in 20f7f2dd71/mypy/sitepkgs.py and realised that everything came down to the Python `site` module and more specifically https://docs.python.org/3.7/library/site.html#site.getsitepackages which in turn relies on https://docs.python.org/3.7/library/site.html#site.PREFIXES .

Our hero created a copy of the environment created by `python.withPackages` and manually modified it to confirm his findings, and it turned out it was indeed the case.
Our hero had damaged the dragon and it was time for a celebration.

He went out and acquired some mead which he ingested while he typed up his story and waited for the dragon to finally die (the commit caused a mass-rebuild, I had to wait for my repro).

In the end all was good in [NixOps-AWS](https://github.com/nixos/nixops-aws)-town and type checks could run. (PR for that incoming tomorrow).
2020-03-14 21:39:31 +00:00
Frederik Rietdijk
13e7a3e112 Python: introduce tests for interpreters
This adds tests to the passthru of all Python interpreters.
2020-03-14 15:05:37 +01:00
Frederik Rietdijk
dbf125d286 Python: introduce tests for interpreters
This adds tests to the passthru of all Python interpreters.
2020-03-14 15:01:30 +01:00
Graham Christensen
39aac70d74 pythonMinimal: don't include site-customise
Experimenting with patching the site-customize file causes mass
rebuilds for no reason.
2020-03-14 15:02:51 +01:00
Frederik Rietdijk
31855d74a3 python3: 3.7.6 -> 3.7.7 2020-03-12 12:00:26 +01:00
Jonathan Ringer
3973a3c79c python: add pythonRemoveTestsDirHook
This will automatically remove a top-level "tests"
directory from being installed.
2020-03-03 20:01:00 +01:00
Jonathan Ringer
fe4580359e python39: 3.9.0.a3 -> 3.9.0.a4 2020-03-03 07:33:39 +01:00
Jonathan Ringer
2a019cc48c python38: 3.8.1 -> 3.8.2 2020-03-03 07:33:39 +01:00
Frederik Rietdijk
79217339d2 Merge master into staging-next 2020-02-29 15:29:11 +01:00
Emily
6d3fc35620 pypy{,3}: use openssl_1_1
"We now support building PyPy with OpenSSL 1.1 in our built-in _ssl
module, as well as maintaining support for previous versions."
-- https://pypy.readthedocs.io/en/latest/release-pypy2.7-v5.6.0.html
2020-02-28 16:06:20 +00:00
Jonathan Ringer
cd97c055a0 python: execute pythonImportsCheckPhase in subshell
Execute in subshell so that the cwd returns back
to the original directory. This is important as
it enables pythonImportsCheck to work with checkPhase
2020-02-19 22:30:50 -08:00
Frederik Rietdijk
38cf6eac19 Merge master into staging-next 2020-02-06 19:43:36 +01:00
Frederik Rietdijk
9827e4994b python39: 3.9.0a2 -> 3.9.0a3 2020-02-06 10:25:18 +01:00
Maximilian Bosch
6b0cd9ad47
Merge branch 'staging' into glibc230
Conflicts:
	pkgs/applications/misc/vit/default.nix
2020-01-28 14:54:51 +01:00
Frederik Rietdijk
dce0ca29d9 Merge master into staging-next 2020-01-28 10:46:13 +01:00
Maximilian Bosch
eddfcc32b4
Merge branch 'staging' into glibc230 2020-01-23 11:31:13 +01:00
Frederik Rietdijk
bc18cc72dd
Merge pull request #77610 from LnL7/darwin-stdenv-python3
darwin-stdenv: bootstrap with python3
2020-01-15 09:24:57 +01:00
Frederik Rietdijk
eba1f79418 pythonPackages.venvShellHook: init
This is a hook that loads a virtualenv from the specified `venvDir`
location. If the virtualenv does not exist, it is created.
2020-01-14 22:36:21 +01:00
Luka Blaskovic
76d90cf1f5 cpython: fetch darwin-libutil.patch to nixpkgs
python3 is now required buildInput for glibc>=2.29.
Remove fetchpatch to solve infinite recursion in
glibc bootstrap process.
2020-01-14 08:26:58 +00:00
Daiderd Jordan
6328518e98
stdenv: bootstrap darwin with python3
- Replaced python override from the final stdenv, instead we
  propagate our bootstrap python to stage4 and override both
  CF and xnu to use it.

- Removed CF argument from python interpreters, this is redundant
  since it's not overidden anymore.

- Inherit CF from stage4, making it the same as the stdenv.
2020-01-13 11:34:36 +01:00
adisbladis
2d6f1ff4dd
python: Add support for installing Python eggs 2020-01-08 13:59:04 +00:00
Jan Tojnar
f4e74edd8c python.pkgs.wrapPython: get rid of warning
When `makeWrapperArgs` variable is not set, `declare -p makeWrapperArgs`
will return with 1 and print an error message to stderr.

I did not handle the non-existence case in b0633406cb
because I thought `mk-python-derivation` will always define `makeWrapperArgs`
but `wrapProgram` can be called independently. And even with `mk-python-derivation`,
`makeWrappers` will not be set unless explicitly declared in the derivation
because of https://github.com/NixOS/nix/issues/1461.

I was lead to believe that because the builds were succeeding and I confirmed
that the mechanism fails when the variable is not defined and `-o nounset` is enabled.
It appears that `wrapPython` setup hook is not running under `-o nounset`, though,
invaldating the assumption.

Now we are checking that the variable exists before checking its type, which
will get rid of the warning and also prevent future error when `-o nounset`
is enabled in the setup hook.

For more information, see the discussion at
https://github.com/NixOS/nixpkgs/commit/a6bb2ede232940a96150da7207a3ecd15eb6328
2019-12-30 16:46:55 +01:00
Jan Tojnar
a6bb2ede23
python.pkgs.wrapPython: fix string makeWrapperArgs
Bash takes an assignment of a string to an array variable:

local -a user_args
user_args="(foo bar)"

to mean appending the string to the array, not parsing the string into
an array as is the case when on the same line as the declaration:

local -a user_args="(foo bar)"

b0633406cb extracted the declaration before
the newly branched code block, causing string makeWrapperArgs being added
to the array verbatim.

Since local is function scoped, it does not matter if we move it inside
each of the branches so we fix it this way.
2019-12-28 22:23:36 +01:00
worldofpeace
4a2621da53
Merge pull request #76283 from jtojnar/python-mwa
python.pkgs.wrapPython: fix makeWrapperArgs
2019-12-26 19:21:37 -05:00
Jan Tojnar
4bbc6cc66f
Merge branch 'staging-next' into staging 2019-12-25 05:18:52 +01:00
Jan Tojnar
ca39dd3a8a
Merge branch 'master' into staging-next 2019-12-25 05:15:06 +01:00
Jan Tojnar
b0633406cb
python.pkgs.wrapPython: fix makeWrapperArgs
When `makeWrapperArgs` is a Bash array, we only passed the first
item to `wrapProgram`. We need to use `"${makeWrapperArgs[@]}"`
to extract all the items. But that breaks the common string case so
we need to handle that case separately.
2019-12-23 18:02:44 +01:00
Frederik Rietdijk
ad733b5505 python37: 3.7.5 -> 3.7.6 2019-12-19 17:37:02 +01:00
Frederik Rietdijk
c0c65fe83c python39: 3.9.0a1 -> 3.9.0a2 2019-12-19 17:36:21 +01:00
Frederik Rietdijk
2012dd5734 python38: 3.8.0 -> 3.8.1 2019-12-19 17:36:21 +01:00
Frederik Rietdijk
5796029c5d python36: 3.6.9 -> 3.6.10 2019-12-19 17:36:20 +01:00
Andreas Rammhold
e9f522eee1
python: remove _manylinux.py
This will turn manylinux support back on by default.

PIP will now do runtime checks against the compatible glibc version to
determine if the current interpreter is compatible with a given
manylinux specification. However it will not check if any of the
required libraries are present.

The motivation here is that we want to support building python packages
with wheels that require manylinux support. There is no real change for
users of source builds as they are still buildings packages from source.

The real noticeable(?) change is that impure usages (e.g. running `pip
install package`) will install manylinux packages that previously
refused to install.
Previously we did claim that we were not compatible with manylinux and
thus they wouldn't be installed at all.

Now impure users will have basically the same situation as before: If
you require some wheel only package it didn't work before and will not
properly work now. Now the program will fail during runtime vs during
installation time.

I think it is a reasonable trade-off since it allows us to install
manylinux packages with nix expressions and enables tools like
poetry2nix.

This should be a net win for users as it allows wheels, that we
previously couldn't really support, to be used.
2019-12-16 16:37:16 +01:00
Tobias Pflug
418ad571c3 Fix manylinux packages
Make sure lib outputs are used where applicable.
2019-12-13 11:40:44 +01:00
Frederik Rietdijk
6530535b20
manylinux packages for Python
This adds three lists with manylinux dependencies as well as three
packages that include all the manylinux dependencies.
2019-12-05 09:56:20 +00:00
Frederik Rietdijk
92d2153e6c pythonInterpreters: remove unnecessary rec 2019-12-02 20:10:51 +01:00
Frederik Rietdijk
1d18c5a0fe Merge staging-next into staging 2019-11-24 10:13:31 +01:00
Frederik Rietdijk
182571cdc3 update-python-libraries: ignore pyproject 2019-11-22 08:37:03 +01:00
Frederik Rietdijk
1939a97811 python3: add pythonForBuild as parameter, fixes python3Minimal
`pythonForBuild` exists for cross-compilation. When one overrides
python, one needs to ensure pythonForBuild matches.
2019-11-21 22:00:23 +01:00
Frederik Rietdijk
ad3ef645b0 python3Minimal: 3.7.4 -> 3.7.5
Base it on python37 so we ensure the package remains up to date.
2019-11-21 15:34:37 +01:00
Frederik Rietdijk
65edeb8633 Merge master into staging-next 2019-11-20 10:01:49 +01:00
Frederik Rietdijk
648152fdbb python39: init at 3.9.0a1
It's a year until the final release but this will give a chance to test
out certain features and how it integrates with other packages.
https://www.python.org/dev/peps/pep-0596/
2019-11-20 09:42:27 +01:00
Jan Tojnar
ae465621ff
pythonPackages.pipBuildHook: fix unbound variable
for compatibility with set -u
2019-11-20 05:06:51 +01:00
Frederik Rietdijk
c4e30cf98c Merge staging-next into staging 2019-11-05 14:18:08 +01:00
Frederik Rietdijk
03a9822405 Merge master into staging-next 2019-11-05 14:17:37 +01:00
John Ericson
acd2d19484
Merge pull request #72347 from NixOS/bash-no-undef-vars
treewide: `set -u` everywhere
2019-11-04 19:52:33 -05:00
Jonathan Ringer
deb201b311 update-python-libraries: update usage comments 2019-11-03 12:03:27 -08:00
Frederik Rietdijk
9d59d57d45 Merge staging-next into staging 2019-11-03 14:01:28 +01:00
Frederik Rietdijk
7827d3f449 python35: 3.5.8 -> 3.5.9
There were no new changes in version 3.5.9; 3.5.9 was released only because of a CDN caching problem,
which resulted in some users downloading a prerelease version of the 3.5.8 .xz source tarball.
Apart from the version number, 3.5.9 is identical to the proper 3.5.8 release.
2019-11-03 11:21:05 +01:00
Dmitry Kalinkin
3466faf9d8 pythonPackages.setuptoolsBuildHook: fix debug message 2019-11-03 10:34:44 +01:00
John Ericson
b7f4bda282 treewide: *Phase(s)? variables are optional
If these aren't defined, the stdenv defaults are used in the `*Phase`
case, or no extra phases are done, in the `*Phases` case.
2019-11-01 14:44:44 -04:00
John Ericson
2811b032d6 treewide: Make still dont* Variables are optional in most cases
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:

 - cc-wrapper's `dontlink`, because it already is handled.

Also, in nix files escaping was manually added.

EMP
2019-11-01 14:44:44 -04:00
Frederik Rietdijk
ccf514a61d python35: 3.5.7 -> 3.5.8 2019-10-29 10:39:50 +01:00
Mario Rodas
0f8dc3cc2e
pypy: fix build on darwin 2019-10-26 04:20:00 -05:00
Frederik Rietdijk
43bbecea5e pythonInterpreters.graalpython37: init
Add an interpreter with package set for graalpython 3.7.
2019-10-22 09:16:18 +02:00
Frederik Rietdijk
5b55013aa2 python2: 2.7.16 -> 2.7.17
Co-authored-by: Dmitry Kalinkin <dmitry.kalinkin@gmail.com>
2019-10-20 19:48:32 +02:00
Frederik Rietdijk
aedf4fc8fc Revert "update-python-libraries: use version key for latest release (#68857)"
This broke support for --target.

This reverts commit e8c29fa77c.
2019-10-18 09:53:55 +02:00
Frederik Rietdijk
7181aca6d9 update-python-libraries: handle other format 2019-10-18 09:53:55 +02:00
Frederik Rietdijk
12441bdf9f Merge staging-next into staging 2019-10-16 11:17:12 +02:00
Frederik Rietdijk
41abe12bb3 python37: 3.7.4 -> 3.7.5 2019-10-16 09:14:48 +02:00
Matthew Glazar
81d15948cc python38: fix build on macOS
Python 3.8 fails to build on macOS for two reasons:

* python-3.x-distutils-C++.patch fails to apply cleanly.
* An #include for <util.h> is missing, causing a build failure:

    ./Modules/posixmodule.c:6586:9: error: implicit declaration of function 'openpty' is invalid in C99
        if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
            ^

Use the correct version of python-3.x-distutils-C++.patch, and add a
patch to #include <util.h>.
2019-10-15 13:12:13 +02:00
Frederik Rietdijk
911b053802 python38: 3.8.0rc1 -> 3.8.0 2019-10-15 11:56:01 +02:00
Jan Tojnar
6c8aed6391
Merge branch 'master' into staging-next 2019-10-12 00:50:21 +02:00
Frederik Rietdijk
1b8f8a65ba python38: 3.8.0b3 -> 3.8.0rc1 2019-10-11 13:57:18 +02:00
Frederik Rietdijk
503081fa5b Merge staging into staging-next 2019-09-29 11:05:22 +02:00
Jon
28af6ac647 python3Packages.flit: fix tests and packaging (#69546)
* python3Packages.flit: fix tests

* python: fix flit setup hook
2019-09-26 21:09:03 +00:00
adisbladis
2b2c6e0bc4
pythonPackages.pipBuildHook: Add wheel dependency
And remove missing "options"
2019-09-26 17:17:16 +01:00
Sebastian Jordan
5505d2f036 python: Fix invalid pip call in setuptoolsShellHook 2019-09-22 09:55:13 +02:00
Mario Rodas
e8c29fa77c
update-python-libraries: use version key for latest release (#68857)
From warehouse API reference [1]:

     GET /pypi/<project_name>/json

     Returns metadata (info) about an individual project at the latest
     version […]

[1] https://warehouse.pypa.io/api-reference/json/#project
2019-09-16 17:34:15 -05:00
Frederik Rietdijk
c99529a4b6 python.pkgs.wheelUnpackHook: propagate wheel
This was accidentally removed when buildPython* was rewritten as hooks.
2019-09-10 22:26:20 +02:00
volth
7b8fb5c06c treewide: remove redundant quotes 2019-09-08 23:38:31 +00:00
Frederik Rietdijk
9894a70299 Merge staging into staging-next 2019-09-06 22:47:40 +02:00
Frederik Rietdijk
66bc7fc1b3 Merge master into staging-next 2019-09-06 22:46:05 +02:00
Frederik Rietdijk
b75f9a5f6d pythonInterpreters.pypy*prebuilt: use openssl 1.0 2019-09-06 21:45:10 +02:00
Frederik Rietdijk
f7e28bf5d8 Split buildPythonPackage into setup hooks
This commit splits the `buildPythonPackage` into multiple setup hooks.

Generally, Python packages are built from source to wheels using `setuptools`.
The wheels are then installed with `pip`. Tests were often called with
`python setup.py test` but this is less common nowadays. Most projects
now use a different entry point for running tests, typically `pytest`
or `nosetests`.

Since the wheel format was introduced more tools were built to generate these,
e.g. `flit`. Since PEP 517 is provisionally accepted, defining a build-system
independent format (`pyproject.toml`), `pip` can now use that format to
execute the correct build-system.

In the past I've added support for PEP 517 (`pyproject`) to the Python
builder, resulting in a now rather large builder. Furthermore, it was not possible
to reuse components elsewhere. Therefore, the builder is now split into multiple
setup hooks.

The `setuptoolsCheckHook` is included now by default but in time it should
be removed from `buildPythonPackage` to make it easier to use another hook
(curently one has to pass in `dontUseSetuptoolsCheck`).
2019-09-06 15:18:45 +02:00
Matthew Bauer
98662eba0e python: enable loadable sqlite extensions (#67472)
Our sqlite supports it, so might as well make use of it.

Fixes #66526
2019-08-31 07:59:00 -04:00
volth
08f68313a4 treewide: remove redundant rec 2019-08-28 11:07:32 +00:00
Frederik Rietdijk
98640fd482 Merge master into staging-next 2019-08-27 16:36:47 +02:00
Marek Mahut
ee62ec62ec python2: CVE-2018-20852
Fixes #67200

(cherry picked from commit 302cac35f5)
2019-08-27 10:04:55 +02:00
Matthew Bauer
48d0e9401e cpython: skip macOS system frameworks
We don’t want cpython picking up /Library/Frameworks and
/System/Library/Frameworks which contains Tcl.framework. Instead it
should use the one provided by Nix. this would not be an issue if
sandboxing was enabled, but unfortunately that has its own issues.

Fixes #66647
2019-08-25 20:35:19 -04:00
Vladimír Čunát
2e6bf42a22
Merge branch 'master' into staging-next
There ver very many conflicts, basically all due to
name -> pname+version.  Fortunately, almost everything was auto-resolved
by kdiff3, and for now I just fixed up a couple evaluation problems,
as verified by the tarball job.  There might be some fallback to these
conflicts, but I believe it should be minimal.

Hydra nixpkgs: ?compare=1538299
2019-08-24 08:55:37 +02:00
Robin Gloster
c7e75df21f
Merge remote-tracking branch 'upstream/master' into openssl-1.1 2019-08-23 02:00:31 +02:00
adisbladis
99e6ca2bdc python3Minimal: Add top-level for a minimal Python 3 build
This builds Python without optional dependencies.

We can't just use python3.override, as things like
python3Minimal.withPackages would pass the wrong python derivation into
these modules.
2019-08-21 14:52:49 +02:00