Cython is a Python compiler that emits native .so modules. By default, python derivations run tests in the wrong directory to see these modules and tests fail.
Issue #255262 documents the root cause and solution for this problem.
This PR adds a description of the problem and the most common solution to the test troubleshooting list.
Replace writeReferencesToFile with writeClosure.
Make writeClosure accept a list of paths instead of a path.
Re-implement with JSON-based exportReferencesGraph interface provided by
__structuredAttrs = true.
Reword the documentation.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Someone Serge <sergei.kozlukov@aalto.fi>
* doc: fix and simplify stylesheets for the manuals, fix nrd bug
* Add anchorjs script to add links on section headers
* Fix another nrd bug, address style changes
* Use span instead of a for inline span syntax
This PR refactor CUDA setup hooks, and in particular
autoAddOpenGLRunpath and autoAddCudaCompatRunpathHook, that were using a
lot of code in common (in fact, I introduced the latter by copy pasting
most of the bash script of the former). This is not satisfying for
maintenance, as a recent patch showed, because we need to duplicate
changes to both hooks.
This commit abstract the common part in a single shell script that
applies a generic patch action to every elf file in the output. For
autoAddOpenGLRunpath the action is just addOpenGLRunpath (now
addDriverRunpath), and is few line function for
autoAddCudaCompatRunpathHook.
Doing so, we also takes the occasion to use the newer addDriverRunpath
instead of the previous addOpenGLRunpath, and rename the CUDA hook to
reflect that as well.
Co-Authored-By: Connor Baker <connor.baker@tweag.io>
Following [Best Practices](https://nix.dev/guides/best-practices#with-scopes),
`with` is a problematic language construction and should be avoided.
Usually it is employed like a "factorization": `[ X.A X.B X.C X.D ]` is written
`with X; [ A B C D ]`.
However, as shown in the link above, the syntatical rules of `with` are not so
intuitive, and this "distributive rule" is very selective, in the sense that
`with X; [ A B C D ]` is not equivalent to `[ X.A X.B X.C X.D ]`.
However, this factorization is still useful to "squeeze" some code, especially
in lists like `meta.maintainers`.
On the other hand, it becomes less justifiable in bigger scopes. This is
especially true in cases like `with lib;` in the top of expression and in sets
like `meta = with lib; { . . . }`.
That being said, this patch removes most of example code in the current
documentation.
The exceptions are, for now
- doc/functions/generators.section.md
- doc/languages-frameworks/coq.section.md
because, well, they are way more complicated, and I couldn't parse them
mentally - yet another reason why `with` should be avoided!
`snapTools.makeSnap` has produced broken snaps since at least Oct 2020,
as indicated by the following issue: https://github.com/NixOS/nixpkgs/issues/100618
No person has shown interest in maintaining it, and given that there is
no fix available, it's assumed that all attempts made to fix that
function have not succeeded.
Given that `snapTools` only contained `makeSnap`, it was removed
completely.
This is an alternative to `fetchNpmDeps` that is notably different in that it uses metadata from `package.json` & `package-lock.json` instead of specifying a fixed-output hash.
Notable features:
- IFD free.
- Only fetches a node dependency once. No massive FODs.
- Support for URL, Git and path dependencies.
- Uses most of the existing `npmHooks`
`importNpmLock` can be used _only_ in the cases where we need to check in a `package-lock.json` in the tree.
Currently this means that we have 13 packages that would be candidates to use this function, though I expect most usage to be in private repositories.
This is upstreaming the builder portion of https://github.com/adisbladis/buildNodeModules into nixpkgs (different naming but the code is the same).
I will archive this repository and consider nixpkgs the new upstream once it's been merged.
For more explanations and rationale see https://discourse.nixos.org/t/buildnodemodules-the-dumbest-node-to-nix-packaging-tool-yet/35733
Example usage:
``` nix
stdenv.mkDerivation {
pname = "my-nodejs-app";
version = "0.1.0";
src = ./.;
nativeBuildInputs = [
importNpmLock.hooks.npmConfigHook
nodejs
nodejs.passthru.python # for node-gyp
npmHooks.npmBuildHook
npmHooks.npmInstallHook
];
npmDeps = buildNodeModules.fetchNodeModules {
npmRoot = ./.;
};
}
```
I was looking at
https://nixos.org/manual/nixpkgs/stable/#buildpythonpackage-parameters to
import a Python package and noticed that the link for the `hooks` in
`pyproject` option is broken due to a typo (used <kbd>0</kbd> instead of
<kbd>)</kbd>).
Signed-off-by: Mihai Maruseac <mihai.maruseac@gmail.com>
Much like the previous commit that adds dependencies &
optional-dependencies this aligns PEP-517 build systems with how they
are defined in PEP-518/PEP-621.
The naming `build-system` (singular) is aligned with upstream Python standards.
Since https://github.com/NixOS/nixpkgs/pull/161835 we've had the
concept of `passthru.optional-dependencies` for Python optional deps.
Having to explicitly put optional-dependencies in the passthru attrset
is a bit strange API-wise, even though it semantically makes sense.
This change unifies the handling of non-optional & optional Python
dependencies using the names established from PEP-621 (standardized pyproject.toml project metadata).
This commit adds support for swapping out the compression algorithm
used in all major docker-tools commands that generate images. The
default algorithm remains unchanged (gzip).
There is an arbitrary mapping being done right now between
nixpkgs lua infrastructre and luarocks config schema.
This is confusing if you use lua so let's make it possible to use the
lua names in the nixpkgs, thanks to the lib.generators.toLua convertor.
The only nixpkgs thing to remember should be to put the config into `luarocksConfig`
`buildLuarocksPackage.extraVariables` should become `buildLuarocksPackage.luarocksConfig.variables`
I believe it would be helpful to better explain how to use
`nuget-to-nix` for those who aren't familar with the .NET ecosystem as I
was personally stumped on how to use it.
This allows for adding new, conditionally set, derivation attributes
to an existing derivation without changing any output paths in the
case where the condition is not met.
`lib.recursiveUpdate` indiscriminately recurses into all attribute sets,
also into derivations. This means that it is possible that evaluating a
derivation in the final haskell package set can cause something in
`prev.haskell` to be forced by `recursiveUpdate`, potentially causing an
evaluation error that should not happen.
It can be fixed using a well-crafted predicate for
`lib.recursiveUpdateUntil`, but most robust is just explicitly writing
out the desired merging manually.
Rename channel `nixpkgs` to `nixpkgs-unstable`. Based on the
[repo branches](https://github.com/NixOS/nixpkgs/branches)
and [channel status](https://status.nixos.org) I don't believe there is
a `nixpkgs` channel. This confused me as a Nixpkgs beginner.
Rename to `nixpkgs-unstable`, which does exist.
Refer to "nixpkgs repository" consitently. Make the capitalization and
"code quoting" consistent when referring to the repository itself.
doc: add figure definition to bespoke syntax reference
doc: add example definition to bespoke syntax reference
doc: add footnote definition to beskpoke syntax reference
The usage of footnotes in the manuals is not the one documented
in markdown-it-py: https://python-markdown.github.io/extensions/footnotes/
doc: add inline comment definition to beskpoke syntax reference
doc: add typographic replacements to beskpoke syntax reference
doc: Fix rendering of bespoke syntax reference
doc: remove references to DocBook in the NixOS manual
doc: add entry on lack of HTML support
doc: Minor improvement
doc: update typographic replacements entry in beskpoke syntax reference
doc: add link reference definitions to beskpoke syntax reference
doc: fix footnote definition in beskpoke syntax reference
doc: Minor improvements from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
* Note on how to use shadowSetup with buildLayeredImage
* Update doc/build-helpers/images/dockertools.section.md
---------
Co-authored-by: Silvan Mosberger <github@infinisil.com>
* doc: add types to template
* mention types explicitly
* use separator that allows for more items in the type declaration
Co-authored-by: Daniel Sidhion <DanielSidhion@users.noreply.github.com>
---------
Co-authored-by: Daniel Sidhion <DanielSidhion@users.noreply.github.com>
* doc: improve documentation for trivial text writing functions
Co-authored-by: Brian Merchant <bzm3r@proton.me>
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Co-authored-by: Alexander Groleau <alex@proof.construction>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Per RFC 9110, [section 8.8.1][1], different representations of the same
resource should have different Etags:
> A strong validator is unique across all versions of all
> representations associated with a particular resource over time.
> However, there is no implication of uniqueness across representations
> of different resources (i.e., the same strong validator might be in
> use for representations of multiple resources at the same time and
> does not imply that those representations are equivalent)
When serving statically compressed files (ie, when there is an existing
corresponding .gz/.br/etc. file on disk), Nginx sends the Etag marked
as strong. These tags should be different for each compressed format
(as shown in an explicit example in section [8.8.3.3][2] of the RFC).
Upstream Etags are composed of the file modification timestamp and
content length, and the latter generally changes between these
representations.
Previous implementation of Nix-specific Etags for things served from
store used the store hash. This is fine to share between different
files, but it becomes a problem for statically compressed versions of
the same file, as it means Nginx was serving different representations
of the same resource with the same Etag, marked as strong.
This patch addresses this by imitating the upstream Nginx behavior, and
appending the value of content length to the store hash.
[1]: https://www.rfc-editor.org/rfc/rfc9110.html#name-validator-fields
[2]:
https://www.rfc-editor.org/rfc/rfc9110.html#name-example-entity-tags-varying
lualatex assumes a writeable font cache relative to `$HOME`, for nix this has two implications.
First, the cache might diverge from the nix store if users use LuaLaTeX.
Second, `$HOME` needs to be set to a writable path in derivations.
All other functions are in the form of `*{c,C}heckpointBuild*`, so we
deprecate the `mkCheckpointedBuild` function in favor of `mkCheckpointBuild`.
Also address some inconsistencies in the docs: some `buildOutput` should
actually be `incrementalBuildArtifacts`.
https://www.pcre.org/
The `pcre` library is "now at end of life, and is no longer being
maintained," according to the upstream maintainers. Accordingly, we
should replace uses of `pcre` with its actively maintained successor,
`pcre2`, wherever possible .
It is fine to use `with` on the inputs, since that increases the
overall readability of the package.
Removes `wheel` from `nativeBuildInputs`, since it is a result of
cargo culting from an earlier setuptools example, that was wrong, and
it is not required, since it is provided by setuptools itself.
The python-updates branch was formerly called python-unstable, but the
new branch name was never mentioned in the docs. This commit changes the
branch name in the docs to python-updates.
We get a dependency list with pub2nix now. We can no longer easily distinguish between development dependency dependencies and regular dependency dependencies, but we weren't doing this anyway.
The mkBinaryCache section now follows the convention of using one
line per sentence, as well as using admonitions for notes and
examples.
The example syntax was updated to define a fully-working nix package.
The following changes are made:
- Document how `wrapType2` and `wrapType1` are the same thing.
- Expand on how `wrapType2` works and additional arguments it uses.
- Document `extract` and show how it's used in combination with
`wrapType2`.
- Provide full working examples using the new admonition syntax.
Fixes some mistakes regarding the references to cargoHash in the codes referenced.
Fixes a typo for cargoSha256.
States that cargoHash should be preferred.
Without this, it's impossible to override the lockFile as the default
overrideAttrs is applied after the composition in buildNimPackage has
read the lock file and generated the nim flags from it.
For the time being, we're moving towards https://nix.dev/ containing
all tutorials and guides. The Nixpkgs manual is reinforced to be a
_reference_ manual. While it's not just reference for now, that's what
the docs team is working towards.
This commits rewrites the Nixpkgs manual introduction to reflect that
and point to some more useful links. The contribution docs are updated
similarly so it's not missed.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
GitHub supported special markdown syntax for emphasising blocks for some
time. This was however a beta feature, and still is, so it's subject to
changes.
Recently such a change happened: The syntax is different now.
See https://github.com/orgs/community/discussions/16925 for more
information
We keep running into situations where we can't get the right
combination of rustc flags through build systems into rustc.
RUSTFLAGS is the only variable supported across build systems, but if
RUSTFLAGS is set, Cargo will ignore all other ways of specifying rustc
flags, including the target-specific ones, which we need to make
dynamic musl builds work. (This is why pkgsCross.musl64.crosvm is
currently broken — it works if you unset separateDebugInfo, which
causes RUSTFLAGS not to be set.)
So, we need to do the same thing we do for C and C++ compilers, and
add a compiler wrapper so we can inject the flags we need, regardless
of the build system.
Currently the wrapper only supports a single mechanism for injecting
flags — the NIX_RUSTFLAGS environment variable. As time goes on,
we'll probably want to add additional features, like target-specific
environment variables.
The Nixpkgs documentation on the linux kernel builders focused on
using and extending kernels that were already packaged, but never
mentioned that it's possible to also build a kernel almost "from
scratch".
The NixOS documentation went a bit deeper on manual linux kernel
configs, but that information wasn't particularly NixOS-specific.
This commit consolidates the information related to building the
kernel on Nixpkgs's documentation, while keeping any additional
NixOS-specific information on NixOS's documentation.
An additional README.md was created for contributor-facing
documentation.
Update instructions for manual build phases execution via `nix-shell` to
cover all phases. There is no easy way of getting those commands, so it
makes a sense to have them all properly documented.
While the word 'simply' is usually added to encourage readers, it often has the
opposite effect and may even appear condescending, especially when the reader
runs into trouble trying to apply the suggestions from the documentation. It is
almost always an improvement to simply drop the word from the sentence.
(there are more possible improvements like this, we can apply those in separate
PRs)
Replaced with python3Packages except for the following to avoid misuse of pythonPackages
1. which we need for the explanation
2. which are replaced by python3Packages as variables
This seems like a pretty easy-to-fix oversight, and it documents the
getExe function, which I never knew about until one day I saw a PR
using it. Let's include it in the manual.
* luarocks-packages-updater: init
Goal is to make it possible to maintain out-of-tree luarocks packages
without needing to clone nixpkgs.
maintainers/scripts/update-luarocks-packages gets renamed to
pkgs/development/lua-modules/updater/updater.py
Once merged you can run for instance
nix run nixpkgs#luarocks-packages-updater -- -i contrib/luarocks-packages.csv -o contrib/generated-packages.nix
I also set the parallelism (--proc) to 1 by default else luarocks fails
because of https://github.com/luarocks/luarocks/issues/1540
* Update maintainers/scripts/pluginupdate.py
Co-authored-by: Marc Jakobi <mrcjkb89@outlook.com>
---------
Co-authored-by: Marc Jakobi <mrcjkb89@outlook.com>
* doc/stdenv: rewrite manual build procedure to be closer to an auto-build
This is based on
<https://jade.fyi/blog/building-nix-derivations-manually/> plus some
more original research.
The previous version of this section did not work for your choice of
simple Haskell package, e.g. haskellPackages.hscolour, due to things
like `compileBuildDriverPhase` and other custom phases that it
does not address at all.
It seems more correct to use genericBuild in development to harmonize it
with what is actually done.
I feel a little bit like I am committing a sin by suggesting using the
experimental CLI in the manual (afaict I am the first to do this), but I
have given the old version of the command, and there are justifiable
reasons to do it:
* The noted limitations with env-vars are fixed. The one with the
non-empty temp directory was one I ran into myself and oh boy was that
not fun to debug.
* Additionally the outputs are set *before* sourcing `setup.sh`: there
is an issue with nix-shell where the original version of `$out` winds
up in `NIX_LDFLAGS` due to _addRpathPrefix, which means that resulting
executables may not run properly.
It is sad that `nix develop` propagates a wrong value of `SHELL` to
builders, though. It is equally sad that `nix-shell` is essentially
abandoned upstream, with undocumented and not insignificant differences
from `nix develop`.
For the exact script differences:
17e6b85d05/src/nix-build/nix-build.cc (L516-L551)db026103b1/src/nix/get-env.sh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Projects building with meson are currently installTargets. Map these
to install tags, which are roughly equivalent. This allows projects to
selectively install components.
I chose not to do nixos.{config, options} because that would make it
look too much like a configuration object, which it is not.
A configuration object I would define as for example the result of
calling NixOS, an attrset with `_type = "configuration";`.
Recreating a configuration object without evalModules is quite
feasible but not guaranteed to be correct, and not maintainable.
Meson now comes with its own set of commands for building, testing,
installing etc., that by default wrap around Ninja. The reason to
prefer using the Meson commands is that they take additional
options (e.g. setting custom timeouts for tests — my motivation for
this change).
Here, I've modified the Meson setup hook so that Meson's test and
install commands will be used instead of Ninja's when Meson's
configurePhase is used. This restriction is to avoid starting to run
Meson directly when dealing with custom build systems that wrap around
Meson, like QEMU's. We don't use meson's compile command, as it just
runs ninja, and that's handled fine by the existing Ninja setup hook.
Naturally the Meson commands don't support entirely the same set of
options that the Ninja ones did, but I checked through Nixpkgs to find
any packages using Meson that used any options that wouldn't be picked
up by this new system. I only found one, and it was just setting
checkTarget = "test", which is the default value for Ninja and has no
Meson equivalent (because we directly tell Meson to run the tests
rather than going through a generic job system like Ninja).
Link: https://github.com/NixOS/nixpkgs/issues/113829
Co-authored-by: Jan Tojnar <jtojnar@gmail.com>
This commit adds minimal documentation of the supported platforms.
More exhaustive documentation would require producing a list of
platforms for each of the 7 tiers. This was attempted in #245368, but it
quickly became clear that that would be a long-term effort.
In the meantime, this commit adds the most important information to the
manual.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
The nixpkgs documentation mentions how to update out of tree plugins but
one problem is that it requires a nixpkgs clone.
This makes it more convenient.
I've had the need to generate vim plugins and lua overlays for other
projects unrelated to nix and this will make updates easier (aka just
run `nix run nixpkgs#vimPluginsUpdater -- --proc=1` or with the legacy commands:
`nix-shell -p vimPluginsUpdater --run vim-plugins-updater`.
I added an optional "nixpkgs" argument to command line parser, which is the path
towards a nixpkgs checkout. By default the current folder.
update-luarocks-packages: format with black
