On Linux the `$TMPDIR` is `/build`. The TMPDIR audit looks for `$TMPDIR`
in the build output, which will then fail with packages like
/buildkite-agent.
This fixes the heuristic to look for `$TMPDIR/` instead.
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit bde99096a8.
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit eb7c50a993.
I originally thought it would just be enough to just check for an INTERP
section in isExecutable, however this would mean that we don't detect
statically linked ELF files, which would break our recent improvement to
gracefully handle those.
In theory, we are only interested in ELF files that have an INTERP
section, so checking for INTERP would be enough. Unfortunately the
isExecutable function is already used outside of autoPatchelfHook, so we
can't easily get rid of it now, so let's actually strive for more
correctness and make isExecutable actually match ELF files that are
executable.
So what we're doing instead now is to check whether either the ELF type
is EXEC *or* we have an INTERP section and if one of them is true we
should have an ELF executable, even if it's statically linked.
Along the way I also set LANG=C for the invocations of readelf, just to
be sure we don't get locale-dependent output.
Tested this with the following command (which contains almost[1] all the
packages using autoPatchelfHook), checking whether we run into any
library-related errors:
nix-build -E 'with import ./. { config.allowUnfree = true; };
runCommand "test-executables" {
drvs = [
anydesk cups-kyodialog3 elasticsearch franz gurobi
masterpdfeditor oracle-instantclient powershell reaper
sourcetrail teamviewer unixODBCDrivers.msodbcsql17 virtlyst
vk-messenger wavebox zoom-us
];
} ("for i in $drvs; do for b in $i/bin/*; do " +
"[ -x \"$b\" ] && timeout 10 \"$b\" || :; done; done")
'
Apart from testing against library-related errors I also compared the
resulting store paths against the ones prior to this commit. Only
anydesk and virtlyst had the same as they didn't have self-references,
everything else differed only because of self-references, except
elasticsearch, which had the following PIE binaries:
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autoconfig
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autodetect
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/categorize
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/normalize
These binaries were now patched, which is what this commit is all about.
[1]: I didn't include the "maxx" package (MaXX Interactive Desktop)
because the upstream URLs are no longer existing and I couldn't
find them elsewhere on the web.
Signed-off-by: aszlig <aszlig@nix.build>
Fixes: https://github.com/NixOS/nixpkgs/issues/48330
Cc: @gnidorah (for MaXX Interactive Desktop)
In strictDeps=false, autoPatchshebangs should use
--build (corresponding to PATH) to lookup commands. This restores the
previous behavior of patchshebangs so that we don’t break stuff that
isn’t careful in the buildInputs vs. nativeBuildInputs distinction.
Unfortunately this won’t work under cross compilation.
Usuage: Add breakpointHook to your `buildInputs` like this:
stdenv.mkDerivation rec {
# ...
buildInputs = [ breakpointHook ];
});
When the build fails as show in this example:
pkgs.hello.overrideAttrs (old: {
buildInputs = [ breakpointHook ];
postPatch = ''
false
'';
});
It will halt execution printing the following message:
build failed in patchPhase with exit code 1
To attach to this build run the following command as root:
cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
Installing cntr and running the command will provide shell access to the
build sandbox of failed build:
sudo cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
WARNING: bad ownership on /nix/var/nix/profiles/per-user/root, should be 1000
[nixbld@localhost:/var/lib/cntr]$
At /var/lib/cntr the sandbox filesystem is mounted. All commands and
files of the system are still accessible within the shell.
To execute commands from the sandbox use the `cntr exec` subcommand.
This causes problems for packages built using a bootstrap stdenv,
resulting in references to /bin/sh or even bootstrap-tools. The darwin
stdenv is much stricter about what requisites/references are allowed but
using /bin/sh on linux is also undesirable.
eg. https://hydra.nixos.org/build/81754896
$ nix-build -A xz
$ head -n1 result-bin/bin/xzdiff
#!/nix/store/yvc7kmw98kq547bnqn1afgyxm8mxdwhp-bootstrap-tools/bin/sh
This reverts commit f06942327a.
The "maxx" package recursively runs isExecutable on a bunch of files and
since the change to use "readelf" instead of "file" a lot of errors like
this one are printed during build:
readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start
While the isExecutable was never meant to be used outside of the
autoPatchelfHook, it's still a good idea to silence the errors because
whenever readelf fails, it clearly indicates that the file in question
is not a valid ELF file.
Signed-off-by: aszlig <aszlig@nix.build>
If the ELF file is not an executable, we do not get a PT_INTERP section,
because after all, it's a *shared* library.
So instead of checking for PT_INTERP (to avoid statically linked
executables) for all ELF files, we add another check to see if it's an
executable and *only* skip it when it is and there's no PT_INTERP.
Signed-off-by: aszlig <aszlig@nix.build>
This hopefully makes patchShebangs respect cross compilation. It
introduces the concept of the HOST_PATH. Nothing is ever executed on
it but instead used as a way to get the proper path using ‘command
-v’. Needs more testing.
/cc @ericson2314 @dtzwill
Fixes#33956Fixes#21138
So far the runtimeDependencies variable has been rather useless unless
you also set dontPatchelf, because the patchelf setup hook ran *after*
the autoPatchelfHook and thus stripped off the additional RPATHs added
using runtimeDependencies.
I did this by moving the autoPatchelfHook to be run in postFixup instead
of fixupOutput, however I needed to replicate the for loop that runs the
hook on all outputs.
Until we have a way to influence order of execution for hooks I've
marked this with an XXX so that we can use fixupOutput again.
Tested this against all packages that use autoPatchelfHook using the
following and checking whether the output contains any errors concerning
shared libraries:
nix-build -E 'with import ./. { config.allowUnfree = true; };
runCommand "test-executables" {
drvs = [
masterpdfeditor franz zoom-us anydesk teamviewer maxx
oracle-instantclient cups-kyodialog3 virtlyst powershell
];
} "for i in $drvs; do for b in $i/bin/*; do \"$b\" || :; done; done"
'
Signed-off-by: aszlig <aszlig@nix.build>
Fixes: https://github.com/NixOS/nixpkgs/issues/43082
Cc: @Ericson2314
A .la file specifies linker flags to link with the library it describes. Its
"dependency_libs" field lists the libraries that this library depends upon.
This list often contains "-l" flags without corresponding "-L" flags. Many
packages in Nixpkgs deal with this in one of these ways:
- delete .la file [1]
- clear dependency_libs [2]
- add -L flags to dependency_libs [3]
- propagate dependencies [4]
Sometimes "dependency_libs" contain wrong "-L" flags pointing to the "dev"
output with headers rather than to the main output with libraries. They have to
be edited or deleted to reduce closure size [5].
Deleting .la files is often but not always safe [6]. Atomatically deleting as
many of them as possible is complex [7]. Deleting .la files that describe
shared rather than static libraries is probably safe; but clearing their
"dependency_libs" field achieves the same effect with less potential for
unintended consequences. This is the approach that may be enabled for all
Nixpkgs.
[1] 2a79d296d3
[2] c83a530985
[3] 9e0dcf3bd9
[4] 01134e698f
[5] f6c73f1e37
[6] https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Handling_Libtool_Archives
[7] https://github.com/gentoo/gentoo/blob/fb1f2435/eclass/ltprune.eclass
If there is a shared object or executable that's using
position-independent code, the file's mime type is
"application/x-pie-executable", so until this change its dependencies
wouldn't be patched.
This simply adds the mime type to the search loop.
Signed-off-by: aszlig <aszlig@nix.build>
I originally wrote this for packaging proprietary games in Vuizvui[1]
but I thought it would be generally useful as we have a fair amount of
proprietary software lurking around in nixpkgs, which are a bit tedious
to maintain, especially when the library dependencies change after an
update.
So this setup hook searches for all ELF executables and libraries in the
resulting output paths after install phase and uses patchelf to set the
RPATH and interpreter according to what dependencies are available
inside the builder.
For example consider something like this:
stdenv.mkDerivation {
...
nativeBuildInputs = [ autoPatchelfHook ];
buildInputs = [ mesa zlib ];
...
}
Whenever for example an executable requires mesa or zlib, the RPATH will
automatically be set to the lib dir of the corresponding dependency.
If the library dependency is required at runtime, an attribute called
runtimeDependencies can be used to list dependencies that are added to
all executables that are discovered unconditionally.
Beside this, it also makes initial packaging of proprietary software
easier, because one no longer has to manually figure out the
dependencies in the first place.
[1]: https://github.com/openlab-aux/vuizvui
Signed-off-by: aszlig <aszlig@nix.build>
Closes: #34506
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
After #31497 starter quoting all values, there arouse the need to left some
values evaluated.
`--set-default var value` expands to `export var=${var-value}`, where value is
not evaluated and literally assigned to var unless it is already set.
`--set-eval var value` expands to `export var=$(eval echo value)`, where value
is evaluated by `eval`.
set-source-date-epoch-to-latest.sh to ignore files newer than "$NIX_BUILD_TOP/.." (unlike "$NIX_BUILD_TOP" it is root-owned and cannot be touched by nixbld1).
`makeWrapper` and `wrapProgram` are being invoked on all kinds of
wacky things (usually with the help of bash globs or other machine
assistance).
So far, I have come across `wrapProgram` being invoked on a directory,
as well as on the empty string.
As far as I can tell, it's only valid to invoke these utilities on a
normal (non-directory, non-device) executable file. This commit
enforces that precondition.
Previously, makeWrapper would accept arguments it didn't recognize,
potentially allowing argument misspellings or broken callers.
Now, makeWrapper dies with a backtrace if it is called incorrectly.
Also changes `wrapProgram` so that it doesn't pass through the first
argument twice --- this was tripping up the argument checking.
Some programs store the executable in a different place and link it
from the `bin` directory. For example, Polari links `$out/bin/polari`
to `$out/share/polari/org.gnome.Polari`. `wrapGAppsHook` did not follow
symlinks so it was not able to wrap Polari, making it unable to access
GObject introspection definitions required for running the program.
I made the wrapping script follow symlinks to fix this corner case.
This setup hook is propagated by gdb. Thus, a typical use is:
$ nix-shell -p gdb nix nix.debug sqlite.debug ...
and gdb will be able find the debug symbols of nix etc. automatically.
As @oxij points out in [1], this breakage is especially serious because
it changes the contents of built environments without a corresonding
change in their hashes. Also, the revert is easier than I thought.
This reverts commit 3cb745d5a6.
[1]: https://github.com/NixOS/nixpkgs/pull/27427#issuecomment-317293040
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.
Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.
Two misc notes:
- `propagated-user-env-packages` also needed to be adjusted as
sometimes it is copied to/from the propagated input files.
- `local fd` should ensure that file descriptors aren't clobbered
during recursion.
This will override the existing winsymlinks setting. nativestrict
will cause ln to fail if it's unable to create a native symlink.
Native symlinks are required for the windows dll loader to find the
libraries.
This script is also used for cross-mingw, but setting CYGWIN
shouldn't cause a problem.
This change fixes several defects in the way `wrapGAppsHook` selected
the executable to wrap.
Previously, it would wrap any top-level files in the target `/bin` and
`/libexec` directories, including directories and non-executable
files. In addition, it failed to wrap files in subdirectories.
Now, it uses `find` to iterate over these directory hierarchies,
selecting only executable files for wrapping.
This change fixes several defects in the way `wrapGAppsHook` selected
the executable to wrap.
Previously, it would wrap any top-level files in the target `/bin` and
`/libexec` directories, including directories and non-executable
files. In addition, it failed to wrap files in subdirectories.
Now, it uses `find` to iterate over these directory hierarchies,
selecting only executable files for wrapping.
Because of bash 4.4 the semantics GLOBIGNORE changed.
This resulted in already compressed manpages to be compressed twice.
Also be careful about symlinks to fix#21777, e.g. the ledger example.
This is required for Aarch64 since a lot of source tarballs ship with
outdated configure scripts that don't recognize aarch64. Simply
replacing the config.guess and config.sub with new versions from
upstream makes them build again.
This same approach is used by at least Buildroot and Fedora. In
principle this could be enabled for all architectures but
conditionalizing this on aarch64 avoids a mass rebuild on x86.
I believe this reduces surprises and is actually simpler semantically.
This is important e.g. for relative symlinks when moving both source
and target - now the order of moving won't matter.
Fixes#20723 (a particular instance of the surprise).
extraFlagsArray should not be exposed outside of `makeWrapper`, it
should only be possible to set it inside a script supplied via the
`--run` argument.
They now go to devman, devdoc, or $outputMan, in that order. This is
to prevent cases such as the man-pages package quietly losing its
section 3 pages.
Prior to this commit, trailing whitespace would be introduced when
modifying '#!' lines with no arguments. For example (whitespace added):
/nix/store/.../foo: interpreter directive changed
from "/bin/bash"
to "/nix/store/...-bash-4.3-p42/bin/bash "
/nix/store/.../bar: interpreter directive changed
from "/bin/baz wef"
to "/nix/store/...-baz wef "
We add a sed command to strip trailing whitespace, so the above commands
would drop the two spaces after "bash", or the one space after "baz wef".
abbradar: fixed commit title
Closes#16785.
The tests need to expand passed variable and very carefully.
I could see no other easy way than to change single-quoting in
makeWrapper to double-quoting.
The tests now fail with the same problem as on master...
find fails when called with an inexistent search path.
That situation may arise when the output is created after by a postFixup hook.
vcunat amended the PR by clarifying one more `return` to `return 0`.
There are no users of it in main tree and recent merge
of multiple outputs branch makes it obsolete for private trees
too.
At the time hook was created, recently merged multiple output
branch was relying on passing flags to autotools to split
outputs, which obviously wasn't working for other build systems
Scatter output was taking different approach where files were
moved out from a build tree based on known paths, which is more
or less what current multiple-outputs.sh hook is able to do too.
For every *.{exe,dll} in $output/bin/ we try to find all (potential)
transitive dependencies and symlink those DLLs into $output/bin
so they are found on invocation.
(DLLs are first searched in the directory of the running exe file.)
The links are relative, so relocating whole /nix/store won't break them.
The hook is activated on cygwin and when cross-compiling to mingw.
- the default --docdir is typically DATAROOTDIR/doc/pkgName
- I saw no other way than to employ some magic to guess this `pkgName`
- user can override it by setting $shareDocName
The importance of glibc makes it worthwhile to provide debug
symbols. However, this revealed an issue with separateDebugInfo: it
was indiscriminately adding --build-id to all ld invocations, while in
fact it should only do that for final links. Glibc also uses non-final
("relocatable") links, leading to subsequent failure to apply a build
ID ("Cannot create .note.gnu.build-id section, --build-id
ignored"). So now ld-wrapper.sh only passes --build-id for final
links.
In some cases the $sourceRoot is missing. Skip the hook instead
of showing the following cryptic error:
find: cannot search `': No such file or directory
/nix/store/0p1afvl8jcpi6dvsq2n58i90w9c59vz1-set-source-date-epoch-to-latest.sh: line 12: [: : integer expression expected
vcunat removed the warning; the hook will just skip silently in these cases.
Perhaps someone can improve on it some time.
For instance, a binary like libfoo.so will cause a symlink
lib/debug/libfoo.so.debug -> .build-id/<build-ID>.debug to be
created. This is primarily useful for use with eu-addr2line, if you
know the name of a binary and the relative address, but not the build
ID.
- fix in silencing some moveToOutput messages
- allow removing (developer) documentation even without defining outputs
(note: some paths are auto-removed by default, e.g. gtk-doc and man3)
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
Now any developer docs are removed by default, unless "docdev"
is in $outputs or $outputDocdev is defined.
Currently devdoc consists of just man3 and gtk-doc.
'[[ ! -v "$propagatedOutputs" ]]' is incorrect and always evaluates to
true. The correct form using double brackets would be
'[[ ! -v propagatedOutputs ]]', but I strongly dislike '[[ ]]' due to
the totally different quoting rules compared to everything else in bash.
You can now pass
separateDebugInfo = true;
to mkDerivation. This causes debug info to be separated from ELF
binaries and stored in the "debug" output. The advantage is that it
enables installing lean binaries, while still having the ability to
make sense of core dumps, etc.
Fixes#9044, close#9667. Thanks to @taku0 for suggesting this solution.
Now we have no modes starting with `/` or `+`.
Rewrite the `-perm` parameters of find:
- completely safe: rewrite `/0100` and `+100` to `-0100`,
- slightly semantics-changing: rewrite `+111` to `-0100`.
I cross-verified the `find` manual pages for Linux, Darwin, FreeBSD.
By default `makeWrapper` will not set argv[0] (this is a reversion to
the old default behavior). Based on the breakage we have seen from
changing the default, this is what most people want. The `wrapProgram`
function will send `--argv0 '"$0"'` to `makeWrapper`, i.e. it will
continue to pass-through the argv[0] that the wrapper is called with.
Now development stuff is propagated from the first output,
and userEnvPkgs from the one with binaries.
Also don't move *.la files (yet). It causes problems, and they're small.
- there were many easy merge conflicts
- cc-wrapper needed nontrivial changes
Many other problems might've been created by interaction of the branches,
but stdenv and a few other packages build fine now.
This is because libxml/libxslt are not the only implementations that
respect $XML_CATALOG_FILES.
Also, look in share/xml for catalogs (in addition to the
now-deprecated xml/dtd and xml/xsl).
This should fix the OpenJDK build, which was failing because paxctl is
in sbin and therefore not automatically added to $PATH.
http://hydra.nixos.org/build/15658346
Now it should contain *all* information from stdenv/setup.sh of
the original mutiple-output branch.
However, the configurability of the output paths is much greater.
This hook allows to scatter files in $out to multiple outputs.
For "bin" and "doc" outputs there are prefefined default masks, but
they can be overriden by setting files_<outname>, for example:
files_bin = [ "/bin/*" "/lib/libexec/" ];
To make an effect hook must be specified in buildInputs.
Stdenv adapters are kinda weird and un-idiomatic (especially when they
don't actually change stdenv). It's more idiomatic to say
buildInputs = [ makeCoverageAnalysisReport ];
Install names need to be absolute paths, otherwise programs that link
against the dylib won't work without setting $DYLD_LIBRARY_PATH. Most
packages do this correctly, but some (like Boost and ICU) do not.
This setup hook absolutizes all install names.
All JARs in $pkg/share/java (for each $pkg in the build inputs) are
added to $CLASSPATH. Thus, you can say
buildInputs = [ setJavaClassPath someJavaDependency ];
and the JARs in someJavaDependency will be found automatically by
tools like javac or ant.
Note that the manual used to say that JARs should be installed in
lib/java; this is now share/java, following the Debian policy:
http://www.debian.org/doc/packaging-manuals/java-policy/x110.html
The directory share/java makes more sense because JARs are
architecture-independent. (Also, a quick grep shows that we were not
exactly consistent about this in Nixpkgs.)
