The setuptools-rust requires some environment variables to really
perform cross build, otherwise it just builds for build platform.
This adds setup hook that introduces these environment variables.
There are three variables.
The PYO3_CROSS_LIB_DIR has to point to the target's Python library
directory. This has to be directory for the target not for the build or
host. We have to choose the correct target Python. I am unsure how to do
that simply in nixpkgs and this this implementations just delays this
and waits for the correct Python when package using this hook is build.
The CARGO_BUILD_TARGET triggers cross compilation in setuptools-rust.
This is simply the Rust target specification.
The CARGO_TARGET_*_LINKER variable should not be essentially required
but setuptools-rust probably mangles the Rust build environment somewhat
and that results to the missing linker. By explicitly specifying it
using the environment variable we force the correct linker.
When installing many python packages, a `direct_url.json` file appeared in the lib directory. Example:
```sh
➤ nix build nixpkgs/29755fec55e58a315b517d431b2be261772f2b80#python3Packages.flask
➤ cat result/lib/python3.10/site-packages/Flask-2.2.3.dist-info/direct_url.json
{"archive_info": {}, "url": "file:///build/Flask-2.2.3/dist/Flask-2.2.3-py3-none-any.whl"}⏎
```
As you can see, that file contains a wrong reference to `/build`.
In https://discuss.python.org/t/pep-610-usage-guidelines-for-linux-distributions/4012/4 there's an explanation on how to avoid this. Here, I'm implementing that change for nixpkgs.
@moduon MT-1075
Include full $name of the derivation into docdir, so documentation for
python package "foo" is installed into $out/share/doc/python3.10-foo-1.2.3
instead of just $out/share/doc/foo, where it may conflict with some
other package named "foo" and not-coinstallable with different
versions of the same python package.
Change from $name to $pname was introduced in [1ee5fca], probably as
unindented side-effect of adding support for formats other than html.
Prior to this commit, pythonRelaxDeps would only support removing
version constraints from "Requires-Dist" lines formatted in a particular
way ("foo (>= 1.2.3)"). This way is deprecated as per PyPA Core Metadata
Specs v2.1 [1]:
> Tools parsing the format should accept optional parentheses around
> this, but tools generating it should not use parentheses.
Additionally, a "Requires-Dist" dependency specification can contain
other metadata than just package name and version (extra names,
environment marker). These were being silently dropped by the prior
version of pythonRelaxDeps, or the version could not be relaxed.
The actual grammar is defined in PEP 508 [2]. Our tool of choice here is
sed extended regexps, so there's only so much we can do to be correct
with this parser. The regexp implemented in this commit makes an attempt
at supporting [extra] names, ; env_markers, as well as version specs
without parentheses. There are still unsupported features (URL specs) as
well as unhandled edge cases, but at some point trying to make the
regexp better is bound to awake ZALGO [3].
[1] https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use
[2] https://peps.python.org/pep-0508/#grammar
[3] https://stackoverflow.com/a/1732454/179806
This adds a test to ensure no new uses of `buildPythonApplication` can
be added to `python-packages.nix`.
Python packages can be grouped into two groups: 1) applications and 2)
packages providing importable modules. In `python-packages.nix` we only
want to have 2). 1) should be in the top-level package set.
To achieve this, all setup hooks need to be marked as being a setup hook.
For the setup hooks in the Python packages set this is done by creating
a new builder, `makePythonHook`.
Because there were issues with splicing, the file importing all the hooks
is converted to an extension. All non-packages were moved out of `python-packages.nix`
into `python-packages-base.nix`. The `keep` argument to `makeScopeWithSplicing
was cleaned up as well; there is no need to keep this one manually in sync
reducing the risk of breaking cross-compilation.
Before packages did not had to set `format = "other";` if had a custom
installPhase and would not produce a dist folder.
The current error message is hard to understand by new users:
> Executing pythonOutputDistPhase
> mv: cannot stat 'dist': No such file or directory
This commit improves the error message.
By default buildPython* runs a hook for detecting conflicting packages.
This hook needs pkg_resources which is part of setuptools.
Before this commit, setuptools was simply added to the build. This meant
that when setuptools was forgotten to be added to the build, the build
and installation would still succeed because of this package from the
hook. During runtime (and cross-compilation) one would notice the
missing package.
Store the intermediate artifacts. In time, we should build, install and
test in separate derivations as that reduces circular dependencies,
avoids rebuilds when fixing tests, and makes it possible to use the
wheels for creating say virtualenv's.
Removes the up until now unused option to specify a `sphinxOutdir` in
favor of allowing to specify multiple builders, which is for example
useful for generating both documentation and manpages using the hook.
Since the output path cannot be determined from within the package we
automatically generate it and add a diversion for manpages, so they land
in the correct output and path.
We have a common pattern here in nixpkgs for Python applications: when a
Python package ships with either a requirements.txt or setup.py file, we
generally end up having to modify its version restriction, otherwise we have
build failures since we package only one specific version of each package
normally.
However, this end up being done in a completely ad-hoc way: some people
use substituteInPlace, some others use sed, others uses patches, etc.
In many cases, the code ends up being buggy, so it may work in one
version and breaks on the next one. We can instead implement one
standard way of doing this, and trying to be a correct as possible.
So this is what this commit does: it implements a new build hook, that
when called will automatically patch the wheel file. This is one of the
most generic ways to patch Python dependencies, and should work in
multiple cases.
Noticed option globbing when tried to enable parallelism by default
locally for most packages by default. python3Packages.yt-dlp failed as:
python3.9-yt-dlp> /nix/store/5mywvxdjkk1q6srwwwgdkzc37ibla801-python3.9-setuptools-61.2.0/lib/python3.9/site-packages/setuptools/dist.py:516: UserWarning: Normalizing '2022.04.08' to '2022.4.8'
python3.9-yt-dlp> warnings.warn(tmpl.format(**locals()))
python3.9-yt-dlp> invalid command name 'build_lazy_extractors--parallel'
The change adds leading whitespace everywhere where options might
already be present.
Pip had the option --build to build in a custom or temporary directory.
Nowadays, pip just listens to TMPDIR, which we already set.
This option was deprecated and is removed in pip 20.3.
