Commit Graph

877 Commits

Author SHA1 Message Date
github-actions[bot]
7aced83bf9
Merge staging-next into staging 2021-01-08 12:29:39 +00:00
Orivej Desh
349585e778 python2: fix ctypes.util.find_library with gcc10
Fixes #108243
2021-01-08 11:19:39 +01:00
Frederik Rietdijk
5c8df2235a python3: don't use sysconfigdataHook on darwin
Resolves issue when building wheels which is a regression introduced by
1a65c5df5f

```
AssertionError: would build wheel with unsupported tag ('cp38', 'cp38',
'darwin_x86_64')
```
2021-01-08 09:34:36 +01:00
Jan Tojnar
0ea0dacc20
Merge branch 'staging-next' into staging 2021-01-07 13:06:41 +01:00
Jan Tojnar
f19eb635b4
Merge branch 'master' into staging-next
b04fc593e7 seems to have accidentally changed mkDerivation function for dfilemanager and solarus-quest-editor so I have reverted that here.
2021-01-07 13:04:31 +01:00
github-actions[bot]
2f7fa4bb88
Merge staging-next into staging 2021-01-05 00:57:02 +00:00
Jonathan Ringer
16174037b3 python38: remove obsolete find_library patch 2021-01-04 15:44:35 -08:00
Frederik Rietdijk
e1ed9b3b08 Revert "python: 3.8.6 -> 3.8.7."
Need to fix a patch first. Next iteration.

This reverts commit 507efdb11f.
2021-01-04 20:58:21 +01:00
Drew Hess
507efdb11f python: 3.8.6 -> 3.8.7. 2021-01-04 20:48:55 +01:00
John Ericson
5c2965145f treewide: Inline more of the static overlay
Picking up where #107238 left off. I think I'll have gotten all the easy
stuff with this.
2021-01-03 21:46:14 +00:00
Ben Wolsieffer
1a65c5df5f cpython: fix finding headers when cross-compiling extension modules 2020-12-28 13:09:00 +01:00
Lucas Ransan
188f1375d8 cpython: fix ctypes.util.find_library 2020-12-28 08:43:32 +01:00
github-actions[bot]
abb757ba28
Merge master into staging-next 2020-12-18 00:42:45 +00:00
Jonathan Ringer
222fafa02b python310: 3.10.0a2 -> 3.10.0a3 2020-12-17 11:18:47 -08:00
Greg Roodt
42d28c2a77 python: 3.9.0 -> 3.9.1 2020-12-12 19:32:08 +01:00
Frederik Rietdijk
76966f8ddd pythonPackages.pip-install-hook: remove --build option
Pip had the option --build to build in a custom or temporary directory.
Nowadays, pip just listens to TMPDIR, which we already set.

This option was deprecated and is removed in pip 20.3.
2020-12-01 14:44:26 +01:00
Frederik Rietdijk
9a63b3d3d6
Merge pull request #104781 from NixOS/staging-next
Staging next
2020-11-30 18:27:29 +01:00
adisbladis
49b8738a5f
python: Propagate packageOverrides to pythonForBuild
When overriding versions of build tools injected via hooks
`packageOverrides` was not taken into account and 2 incompatible
versions of the same package (`wheel` in this case) ended up in the
closure, causing the builds to fail.
2020-11-30 11:12:59 +01:00
Frederik Rietdijk
0d8491cb2b Merge master into staging-next 2020-11-29 13:51:10 +01:00
Frederik Rietdijk
455774e546 Python: add todo note regarding pythonForBuild 2020-11-28 17:36:23 +01:00
Frederik Rietdijk
5804c8a7f5 Python splicing: splice package sets, not Python derivation
In the original commit the various builds of Python were added to
`otherSplices`, instead of the intended Python package sets.
2020-11-28 17:36:23 +01:00
Frederik Rietdijk
6cf25f9dbd Python: rename parameters and arguments passed to passthru
As part of the splicing the build/host/target combinations of the interpreter
need to be passed around internally. The chosen names were not very clear,
implying they were package sets whereas actually there were derivations.
2020-11-28 17:36:23 +01:00
Frederik Rietdijk
8220b0449c python.tests: use self.callPackage instead of super.callPackage
super was incorrectly possible until https://github.com/NixOS/nixpkgs/pull/104201
got merged.
2020-11-28 16:57:07 +01:00
Frederik Rietdijk
cce2fd547b Python: use pythonPackagesBuildHost instead of pythonForBuild
Follow-up to #104201, related to #105113.
2020-11-28 16:36:03 +01:00
Jonathan Ringer
0a4575a1da python/hooks/pythonNamespaces: fix __pycache__ being empty, or not existing 2020-11-26 13:51:05 -08:00
Frederik Rietdijk
0a12b8a5e9 Revert "Revert "Revert "cpython: fix finding headers when cross-compiling extension modules"""
This still does not function without issues. E.g., bootstrapped-pip fails with Python 3.6 and 3.7
as well as 3.8 on 32-bit.

Because this is a stdenv-rebuild it needs to be tested significantly better

This reverts commit 6100bc29f7.
2020-11-25 10:34:23 +01:00
Jonathan Ringer
f5d9dd3050 python/hooks/pythonNamespaces: fix __pycache__ logic 2020-11-23 08:33:04 -08:00
Frederik Rietdijk
95d9ff16f3 python.tests: not when cross-compiling
Running the tests is not possible when cross-compiling.
2020-11-23 15:29:23 +01:00
Frederik Rietdijk
6100bc29f7 Revert "Revert "cpython: fix finding headers when cross-compiling extension modules""
Moving the sysconfig file caused spidermonkey_78 to fail to build. We now symlink it instead.

This reverts commit 6a23dde75b.
2020-11-23 15:29:23 +01:00
Frederik Rietdijk
7726f81b13 python.tests: use self.callPackage instead of super.callPackage
super was incorrectly possible until https://github.com/NixOS/nixpkgs/pull/104201
got merged.
2020-11-22 13:17:24 +01:00
Frederik Rietdijk
6a23dde75b Revert "cpython: fix finding headers when cross-compiling extension modules"
This breaks virtualenv https://github.com/NixOS/nixpkgs/issues/104483.
We should probably not move `_sysconfigdata` but just copy it for cross to
another place or output.

This reverts commit 11806b6ede.
2020-11-22 13:13:23 +01:00
Jonathan Ringer
8d210e2ea4 python3.pkgs.pipInstallHook: don't warn on script installation 2020-11-20 12:25:52 -08:00
Frederik Rietdijk
27d3a18ac7 buildPython*: don't catch conflicts when cross-compiling
Hack until we fix PYTHONPATH.
2020-11-19 20:59:16 +01:00
Ben Wolsieffer
11806b6ede cpython: fix finding headers when cross-compiling extension modules 2020-11-19 20:42:58 +01:00
Frederik Rietdijk
ea7b8978ef Merge master into staging-next 2020-11-19 20:08:15 +01:00
John Ericson
b57c5d4456 python: Use makeScopeWithSplicing
Now non-`buildInputs` that are python packages should be resolved
correctly.
2020-11-19 11:58:07 -05:00
John Ericson
330d468002 python: Use packages from previous stage in setup-hooks
This makes sense as these are tools we want to run at build time.
2020-11-19 11:58:03 -05:00
Frederik Rietdijk
da12fc6838 Merge staging-next into staging 2020-11-18 15:36:56 +01:00
Bernardo Meurer
7edf1ced82 pypy: enable aarch64-linux platform 2020-11-18 09:55:33 +01:00
Anders Kaseorg
9e09ad5182 pypy, pypy3: 7.3.1 → 7.3.2
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-11-18 00:21:41 -08:00
Anders Kaseorg
b7836e5ca6 pypy, pypy3: Follow Heptapod move for download URL
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-11-18 00:21:41 -08:00
Frederik Rietdijk
9fc4a4db5a Merge staging-next into staging 2020-11-13 20:59:08 +01:00
Frederik Rietdijk
3fceafce1e buildPython*: remove pythonRecompileBytecodeHook as dependency
This hook was added to get reproducible bytecode. Because it was causing
issues it was disabled, but still kept as a dependency. Now the main
issue with bytecode reproducibility has been resolved by updating pip to
20.2.4, we remove this hook as a dependency.

If a package with Python code is not yet reproducible, one could add
this hook to `nativeBuildInputs`.
2020-11-12 19:32:30 +01:00
Jonathan Ringer
5a5122418a python310: 3.10.0a1 -> 3.10.0a2 2020-11-11 23:13:44 -08:00
Frederik Rietdijk
20f001c01e Merge master into staging-next 2020-11-09 14:33:52 +01:00
Samuel Gräfenstein
22b73d1282 pkgs/*: add final newline to .nix files 2020-11-09 11:39:28 +00:00
Frederik Rietdijk
61a04f735a python3.pkgs.pip: 20.1.1 -> 20.2.4
Reproducible builds of pyproject projects using pip is resolved.

Fixes https://github.com/pypa/pip/issues/7808
Fixes https://github.com/NixOS/nixpkgs/issues/81441

The more recent c409f69480 caused trouble
with pyproject troubles and had to be reverted anyway.
https://github.com/NixOS/nixpkgs/pull/102222#issuecomment-722380794

Revert "pythonPackages.pip: make reproducible (#102222)"

This reverts commit c409f69480.

Revert "python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR"

This reverts commit aedbade43e.
2020-11-05 15:07:21 +01:00
Frederik Rietdijk
6ddd6b4914 Revert "Merge pull request #93083 from risicle/ris-cpython-debug"
This increase the closure size by 30 MB.

https://github.com/NixOS/nixpkgs/pull/93083#issuecomment-718025366

This reverts commit 4b340cbbb4, reversing
changes made to 47e499f7a4.
2020-11-03 12:50:05 +01:00
zimbatm
c409f69480
pythonPackages.pip: make reproducible (#102222)
The previous attempt wasn't covering all of the bases. It relied on
invoking that pip-install-hook, and didn't apply to pip itself.

The core issue is that the generated .pyc files embed some of the
temporary paths, which are randomly generated. See
https://r13y.com/diff/bf8c3ca3148ebff9ecf41f294cc60b9f209c006d49699e356969ff32d736f1c6-8806a7cca91fdd300e48736bfcd57c4d0b54c1cc2fd61609f35143170862b59c.html

In this new attempt, the approach is to patch the TempFile
implementation directly, so that it creates stable temporary
directories. We also assume that if SOURCE_DATE_EPOCH is set, we are in
a scenario where reproducible builds are desirable and enter that
branch.

See also https://github.com/pypa/pip/issues/7808
2020-10-31 21:24:36 +01:00
Jonathan Ringer
67d2de5ffa python310: init at 3.10a1 2020-10-19 09:31:27 -07:00
Jonathan Ringer
9c9418151d update-python-libraries: allow for top-level attrs to be updated with github fetcher 2020-10-15 12:10:08 -07:00
Frederik Rietdijk
9e1943edc0 Merge master into staging-next 2020-10-13 19:34:34 +02:00
Jonathan Ringer
2a9c3ec083
update-python-libraries: make script updateScript friendly 2020-10-11 12:03:18 -07:00
Frederik Rietdijk
692d219a93 Merge staging-next into staging 2020-10-06 10:25:58 +02:00
Jonathan Ringer
4eaf246f24 update-python-libraries: add github fetcher 2020-10-06 00:55:43 -07:00
Raphael Borun Das Gupta
3696bc0e89 python39: 3.9.0rc2 -> 3.9.0 2020-10-06 08:32:37 +02:00
Jan Tojnar
32b4375f10
Merge branch 'staging-next' into staging 2020-09-29 00:12:29 +02:00
Frederik Rietdijk
7c411b3729 python38: 3.8.5 -> 3.8.6 2020-09-27 13:07:49 +02:00
Frederik Rietdijk
74f3e9cce1 python39: 3.9.0rc1 -> 3.9.0rc2 2020-09-27 11:48:00 +02:00
Johannes Rosenberger
7abb57c7b5 fix passing qt5 version to pythonInterpreters
fixes c88f3adb17, which resulted in
qt 5.15 being used in pythonPackages, despite 5.14 being
declared, and adapts qutebrowser accordingly.

'callPackage { pkgs = pkgs // { … }; }' does not work, because
it does not take into account the recursive evaluation of nixpkgs:

`pkgs/development/interpreters/python/default.nix` calls
`pkgs/top-level/python-packages.nix` with `callPackage`.
Thus, even if the former gets passed the updated `pkgs`,
the latter always gets passed `pkgs.pkgs`.

For the change in the qt5 version to apply consistently, 'pkgs.extend'
must be used.

qutebrowser only used the right qt5 version (5.15) because all
pythonPackages used it anyway.
2020-09-22 21:03:43 +02:00
Drew Risinger
60df61b67e mk-python-derivation: document pyproject format
Optional setting of format == "pyproject", "egg" had been documented
in the manual, but they weren't listed in the function header for
``mk-python-derivation.nix``.
2020-09-16 17:38:41 +02:00
Frederik Rietdijk
af81d39b87 Merge staging-next into staging 2020-09-04 20:03:30 +02:00
Jonathan Ringer
762434fb02 python39: 3.9.0b5 -> 3.9.0rc1 2020-09-03 23:36:28 -07:00
Frederik Rietdijk
e29c1e42e0 Merge staging-next into staging 2020-08-31 19:46:33 +02:00
Frederik Rietdijk
5d8dd5c259 python.tests: don't mix the two types of tests
The integration tests code was included in the environment tests. After
this commit it is hopefully clearer what belongs together.
2020-08-30 12:59:24 +02:00
Frederik Rietdijk
2248a7422f python36: 3.6.11 -> 3.6.12 2020-08-30 08:59:42 +02:00
Spencer Baugh
780caf6690 python37: 3.7.8 -> 3.7.9
Release notes:
https://docs.python.org/release/3.7.9/whatsnew/changelog.html

Most notable is the fixed behavior of asynchronous generators; without
this fix, they fail to work in some scenarios.
2020-08-30 08:52:06 +02:00
Jan Tojnar
91104b5417
Merge branch 'master' into staging-next 2020-08-23 02:00:50 +02:00
Lassulus
7d03cf2c8d
Merge pull request #91667 from DavHau/fix-indentation
mk-python-derivation.nix: fix indentation
2020-08-21 17:07:44 +02:00
Frederik Rietdijk
0ac85bc455 Merge master into staging-next 2020-08-17 14:54:39 +02:00
Alvar Penning
b7ce309e6c pythonFull: add BlueZ support
This commit introduces two changes.

First, cpython gets optional BlueZ support, which is needed for
AF_BLUETOOTH sockets. Therefore bluezSupport was added as a parameter.

Second, the call to the pythonFull packages has been adjusted. The
Python packages have a self-reference called self. This was not adjusted
for the override. As a result, Python packages for this special version
of Python were not built with the overridden Python, but with the
original one.
2020-08-15 18:08:20 +02:00
Frederik Rietdijk
cfe6081cee Merge staging-next into staging 2020-08-15 09:12:42 +02:00
Frederik Rietdijk
4b340cbbb4
Merge pull request #93083 from risicle/ris-cpython-debug
cpython: add separateDebugInfo, allow use of gdb libpython
2020-08-15 08:52:42 +02:00
Daniël de Kok
900b2d5a6c python3: fix impure /bin/sh call in subprocess 2020-08-15 08:34:02 +02:00
Jonathan Ringer
91bec61635
python35: remove 2020-08-12 21:57:18 -07:00
Jan Tojnar
11da469fa5
Merge branch 'staging-next' into staging 2020-08-11 16:18:42 +02:00
Christian Kauhaus
a14859c686 python: Apply patch for CVE-2019-20907
Incluing the patch file in-tree because the upstream patch is not
intended to apply for Python 2.

Re #94004
2020-08-11 16:05:43 +02:00
Anders Kaseorg
b31e4a20a4 pypy, pypy3: 7.1.1 -> 7.3.1
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-10 15:42:42 -07:00
Anders Kaseorg
6f2ec6d967 pythonInterpreters.pypy{27,36}_prebuilt: 7.1.1 -> 7.3.1
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-10 15:42:42 -07:00
Matthew Bauer
d0677e6d45 treewide: add warning comment to “boot” packages
This adds a warning to the top of each “boot” package that reads:

  Note: this package is used for bootstrapping fetchurl, and thus cannot
  use fetchpatch! All mutable patches (generated by GitHub or cgit) that
  are needed here should be included directly in Nixpkgs as files.

This makes it clear to maintainer that they may need to treat this
package a little differently than others. Importantly, we can’t use
fetchpatch here due to using <nix/fetchurl.nix>. To avoid having stale
hashes, we need to include patches that are subject to changing
overtime (for instance, gitweb’s patches contain a version number at
the bottom).
2020-07-31 08:56:53 +02:00
Mario Rodas
ced8ec8488 python39: 3.9.0a4 -> 3.9.0b5 2020-07-26 18:45:54 +02:00
Mario Rodas
79590e27d4 python38: 3.8.3 -> 3.8.5 2020-07-26 18:45:54 +02:00
Robert Scott
ede2e00c9f cpython: expose gdb libpython.py
used together with cpython's debugging symbols, this allows inspection of
the python stack of cpython programs in gdb. this file is a little
different from the rest of the python output by this package, in that it's
not intended to be run by the current python being built, instead by the
python being used by the gdb in question, which could be very different.
therefore placed in its own, but hopefully logical & predictable location.
2020-07-15 19:39:19 +01:00
Robert Scott
6d19ab339a cpython: set separateDebugInfo 2020-07-13 22:30:55 +01:00
Frederik Rietdijk
7935bf793e python37: 3.7.7 -> 3.7.8 2020-07-04 18:13:44 +02:00
Frederik Rietdijk
60fb772ae2 python36: 3.6.10 -> 3.6.11 2020-07-04 18:13:44 +02:00
misuzu
aedbade43e python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR 2020-07-02 17:27:13 +02:00
DavHau
d1c1a0c656 fix indentation in mk-python-derivation.nix 2020-06-27 14:26:03 +00:00
Frederik Rietdijk
f03734efa0 python3: improve cross-compilation of extension modules, fixes #91171 2020-06-21 16:29:57 +02:00
Frederik Rietdijk
b7aec77a7b buildPythonPackage: don't recompile bytecode
There are too many regressions. Instead of reverting all the work that has been
done on this so far, let's just disable it Python-wide. That way we can
investigate and fix it easier.
2020-06-17 14:54:03 +02:00
Jonathan Ringer
b6654d9d6c python2Packages: default to dontUsePythonRecompileBytecode = true; 2020-06-16 13:02:50 -07:00
Frederik Rietdijk
2e4b4e3300 Merge staging-next into staging 2020-06-13 11:03:26 +02:00
Frederik Rietdijk
818cf7827b buildPythonPackage: recompile bytecode for reproducibility
Due to a change in pip the unpacked wheels are no longer reproducible.
We recompile the bytecode to cleanup this error.
https://github.com/NixOS/nixpkgs/issues/81441
2020-06-13 10:36:28 +02:00
Frederik Rietdijk
a7ea828f09 pythonRemoveBinBytecodeHook: fix explanation 2020-06-13 10:20:14 +02:00
Frederik Rietdijk
4087d3fe41 python: don't use optimizations on Darwin
Also, don't use autoreconfHook on Darwin with Python 3.
Darwin builds are still impure and fail with

    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: object file (/nix/store/0lsij4jl35bnhqhdzla8md6xiswgig5q-Libsystem-osx-10.12.6/lib/crt1.10.6.o) was built for newer OSX version (10.12) than being linked (10.6)
    DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
    if test $? -ne 0 ; then \
            echo "generate-posix-vars failed" ; \
            rm -f ./pybuilddir.txt ; \
            exit 1 ; \
    fi
    /nix/store/dsb7d4dwxk6bzlm845z2zx6wp9a8bqc1-bash-4.4-p23/bin/bash: line 5: 72015 Killed: 9               DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars
    generate-posix-vars failed
    make: *** [Makefile:592: pybuilddir.txt] Error 1
2020-06-12 18:29:08 +02:00
Daniël de Kok
1e2b6695cf pythonPackages.setuptoolsBuildHook: do not build in an isolated environment
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:

https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support

For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.

This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
2020-06-06 10:05:26 +02:00
Daniël de Kok
e2309df85e pythonPackages.pipBuildHook: do not build in an isolated environment
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:

https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support

For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.

This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
2020-06-06 10:05:26 +02:00
Frederik Rietdijk
1c68570ab2 Merge staging-next into staging 2020-06-05 19:42:16 +02:00
Frederik Rietdijk
43f71029cc Merge master into staging-next 2020-06-05 19:40:53 +02:00
Frederik Rietdijk
913bee36ed python3Minimal: override python38, not python3
This avoids an infinite recursion, accidentally introduced in b7ff746540.
2020-06-05 16:46:40 +02:00