When 9.2.1 was [released], I apparently was confused by the wording. The
NCG (-fasm) codegen backend for aarch64 not only works on
aarch64-darwin, but also aarch64-linux. `useLLVM` being enabled on
aarch64-linux had no adverse effect, as GHC used -fasm anyways, but it
did inflate closure size unnecessarily which we can rectify now.
[released]: https://www.haskell.org/ghc/blog/20211029-ghc-9.2.1-released.html
This was already applied to GHC 9.2.x, but was not copied to GHC 9.4.x.
I have had issues with this locally.
The same patch works for both Cabal 3.6 and 3.8, so we can just reuse it.
The `meta` set of the binary GHCs is mostly independent of the used
bindist (except for `pname` which includes `variantSuffix`). Thus we
should make sure it can be evaluated even if no bindist is available for
the platform, i.e. evaluating `outPath` may cause an evaluation failure,
but `meta.platforms` not. Use case at present is to make
`lib.meta.availableOn` work everywhere for any GHC (the normal GHCs
inherit their platforms list from their respective boot compiler, at
least for now).
To fix this we need to make sure that shallowly evaluating `passthru`
doesn't force `binDistUsed`, since `mkDerivation` needs to merge
`passthru` into the resulting derivation attribute set, thus forcing the
attribute names of `passthru`. We can easily do this by accessing what
we want to learn from `ghcBinDists` manually and using `or` to fall back
to a sensible default.
The switch to cctools-llvm made several LLVM tools the default on
Darwin, which includes llvm-ar. GHC will try to use `-L` with `ar` when
it is `llvm-ar`, but that doesn’t work currently on Darwin.
See https://gitlab.haskell.org/ghc/ghc/-/issues/23188.
This saves just enough space on aarch64-linux so that the hadrian built
GHCs are under the 3GB Hydra output limit:
| compiler | before | after | Δ |
|----------|------------|------------|------------|
| ghc962 | 3241234736 | 2810740560 | -430494176 |
| ghcHEAD | 3341288328 | 2902760872 | -438527456 |
The total output size can be calculated using (don't forget to use
aarch64-linux):
```
nix-build -A <compiler> | xargs nix path-info -s | awk '{ s += $2 }; END { print s }'
```
haskell.compiler.ghc8102BinaryMinimal: remove at 8.10.2
haskell.compiler.ghc8107BinaryMinimal: remove at 8.10.7
haskell.compiler.ghc924BinaryMinimal: remove at 9.2.4
On aarch64-linux the binary GHCs take up about 2.6GB (which compresses
pretty well on zfs as it turns out), so they are below the output limit
of Hydra. This allows us to drop the special casing of aarch platforms
in haskell-packages.nix. While we're at it, drop the minimal variants so
we don't unnecessarily build variants of the binary GHCs.
The main problem was GHC exceeding the Hydra output limit with profiling
libs on aarch64-linux which made us disable the feature. Nowadays the
limit is 3GB, the GHC output is a bit over 2GB, so easily under the
limit.
aarch64-darwin uses a different codegen backend and was never really
affected by the problem: Its output with profiling enabled is around
1.6GB.
Consequently we can enable profiling for all platforms again, as we have
no output size issues for those we build on Hydra.
Thanks to flokli for helping me track down these up to date numbers.
Hadrian (the GHC build tool) is built separately from GHC. This means
that if `haskell.compiler.ghc961` is overridden to add patches, those
patches will _only_ be applied to the GHC portion of the build, and not
the Hadrian build. For example, backporting this patch to GHC 9.6.1
failed because the changes to `hadrian/` files were not reflected in the
Nix build:
5ed77deb1b
By lifting `src` and `hadrian` from variables defined in the function
body to parameters with default values, the `hadrian/` files can be
overridden using the `haskell.compiler.ghc961.override` function. For
example:
self.haskell.compiler.ghc961.override {
# The GHC 9.6 builder in nixpkgs first builds hadrian with the
# source tree provided here and then uses the built hadrian to
# build the rest of GHC. We need to make sure our patches get
# included in this `src`, then, rather than modifying the tree in
# the `patchPhase` or `postPatch` of the outer builder.
src = self.applyPatches {
src = let
version = "9.6.1";
in
self.fetchurl {
url = "https://downloads.haskell.org/ghc/${version}/ghc-${version}-src.tar.xz";
sha256 = "fe5ac909cb8bb087e235de97fa63aff47a8ae650efaa37a2140f4780e21f34cb";
};
patches = [
# Enable response files for linker if supported
(self.fetchpatch {
url = "5ed77deb1b.patch";
hash = "sha256-dvenK+EPTZJYXnyfKPdkvLp+zeUmsY9YrWpcGCzYStM=";
})
];
};
}
Note that we do have to re-declare the `src` we want, but I'm not sure
of a good way to avoid this while also sharing one set of patches
between the GHC and Hadrian builds.
We previously thought that we only need python if we were going to run
./boot or using emscripten which implements all its wrappers in
python (and likes to reinvoke them). As it turns out, though, hadrian
likes to invoke python itself for generating certain headers of rts
using a script shipped with the GHC source. This fact was obscured
before, since (presumably) sphinx would propagate python into PATH.
Due to link time dead code elimination not working on aarch64-darwin,
some unused store path references in Paths_* modules are retained. This
causes reference cycles when a separate `bin` output is used.
To prevent this, add a patch to Cabal as shipped by GHC which infers
based on the installation layout (which is influenced by
enableSeparateBinOutput, enableSeparateDataOutput etc. in a Nix build)
which references can be retained without causing a reference cycle. This
ensures that packages that were fine with a bin output will also work on
aarch64-darwin. Packages that cause a reference cycle anyways (by
actually using references that do cause one) fail due to a missing
symbol – here we are trading the overall benefit for a more confusing
error message.
For details, refer to the explanation comment in the patch.
`*.*.rts.*.opts` is actually copied from the migration GHC blog post,
but does not, actually, parse: The format is
`<stage>.<package>.<program>.<filetype>.<setting>`, so it would need to
be `*.rts.ghc.opts`. This is already achieved by the broader rule on the
next line.
- Christmas is over!
- Upstream has changed the name of the target triplet used for the JS
backend from js-unknown-ghcjs to javascript-unknown-ghcjs, since Cabal
calls the architecture "javascript":
6636b67023
Since the triplet is made up anyways, i.e. autoconf does not support
it and Rust uses different triplets for its emscripten backends, we'll
just change it as well.
- Upstream fixed the problem with ar(1) being invoked incorrectly by stage0:
e987e345c8
Hadrian does this automatically unfortunately, but unless we correctly
set enableShared as well, mkDerivation will try building shared libs
which will inevitably fail due to missing shared core packages.
Let's stay away from fully_static which does a lot of funky stuff and
was not working before anyways for pkgsStatic.
There is a code generation bug in Cabal-3.6.3.0. For packages configured with
--enable-relocatable, Cabal would generate code that doesn't compile.
There isn't an upstream issue, but the issue is described in the commit that
fixed it:
6c796218c9
It was fixed in Cabal-3.8.*
Backport the fix to the Cabal library that ships with ghc-9.4.4
Cabal 3.8 ships with ghc-9.6, so when 9.6 is released this fix shouldn't be
necessary.
`ghc ? hadrian` can be used to check if a GHC was built using hadrian.
This is often relevant since hadrian changed the ghc libdir location, so
we need to install libs to a different location as well.
GHC ships a [modified] config.sub so that js-unknown-ghcjs is accepted
by autotools. For some platforms, we automatically update config.sub
from upstream's source in order to prevent that builds fail when we use
an outdated config.sub. In this case of course the perfectly up to date
config.sub would reject the target platform we are trying to use, so we
must disable this mechanism for now.
I have asked in the GHC IRC channel if there are any plans on
upstreaming the platform. It would be nice if were able to drop this
change in the future.
This is now possible by building a cross compiler for js-unknown-ghjs
using `pkgsCross.ghcjs.buildPackages.haskell.compiler.ghcHEAD`.
To allow this, the following things needed to be done:
* Disable dependencies that wouldn't work:
- Don't pull in ncurses for terminfo
- Don't pull in libffi
- Don't pull in libiconv
- Don't enable the LLVM backend
- Enable gmp-less native-bignum backend
* Use emscripten instead of a C compiler. The way this works is inspired
by emscriptenPackages, but avoids the following flaws:
- Instead of using a custom configurePhase, just set
`configureScript = "emconfigure ./configure";` which is much simpler.
- Create writable EM_CACHE before configuring, as configure scripts
want to compile test programs.
Additionally, we need to disable the targetCC check, as it is not
applicable with emscripten which never appears as part of stdenv.
* Use generic $configureScript in installPhase to be able to work with
our emconfigure trick.
Note that the corresponding Haskell package set does not work yet. Cabal
doesn't seem to like GHC 9.7 yet and the generic-builder is clueless
about the JS backend.
gmp is part of buildInputs _and_ depsTargetTarget, so we need to check
the host and target platform to be correct. In practice this doesn't
change much though, as gmp.meta.platforms is _quite_ liberal.
Finally building a cross compiler using hadrian is possible, but there
are some outstanding issues regarding external libraries in the package
db which causes issues with ghc-bignum.
There is a code generation bug in Cabal-3.6.3.0. For packages configured with
--enable-relocatable, Cabal would generate code that doesn't compile.
There isn't an upstream issue, but the issue is described in the commit that
fixed it:
6c796218c9
It was fixed in Cabal-3.8.*
Backport the fix to the Cabal library that ships with ghc-9.2.4
Co-authored-by: sternenseemann <sternenseemann@systemli.org>
Closes#200063.
Initial port of our GHC Nix expressions to the new hadrian build system,
as it has become required after 9.4. Unfortunately there are some
regressions affecting us, namely the inability to install a GHC
cross-compiler at the moment (see issue linked in relevant error
message). This means that a lot of specific configuration snippets for
cross-platforms and static compilation have been ported from make
speculatively, as we are unable to test them for the moment.
This shortens the bootstrap chain for 9.4.1 and should be kinder on
rebuilds. It requires some messing around in the configure file, since
it is not officially supported by upstream (but known to work). For now
it saves us the hassle of adding another bindist to nixpkgs. When we
support hadrian, we'll be able to use the already packaged 9.2.2
bindist.
Since the musl / alpine bindist now uses the GMP backend, we need to
learn to tell Hadrian bindists about GMP. Hadrian bindists no longer
have the buildinfo files, instead we need to patch the package db before
installing and recache it afterwards which is not too hard, luckily.
Same goes for libiconv and base as well as libffi and rts on
darwin (those bindists are all produced using hadrian).
See also: https://gitlab.haskell.org/ghc/ghc/-/issues/21554#note_431000
Note that pkgsMusl.haskell.compiler.ghc922Binary still has severe
issues: It can't produce shared libraries because the bindist ships
none (and using the GMP backend has a hard requirement for shared
objects, apparently) and ghci segfaults for unknown reasons at the
moment. However, I've successfully compiled hadrian with it so far, so
perhaps it's good enough.
This was disabled basically by accident before.
The links are jacked, but that was is true for every package; it is not
unique to this PR. I fixed it upstream here:
https://github.com/haskell/haddock/pull/1482
but it's not in any release distributions yet I don't think.
Fixes#171841
Otherwise attempt to build ghcHEAD from local checkout fails as:
$ nix build -L --impure --expr 'with import ~/nm {}; haskell.compiler.ghcHEAD.overrideAttrs (oa: { src = ./.; patches = []; nativeBuildInputs = oa.nativeBuildInputs ++ [ git ]; })' --keep-failed
...
ghc> checking C++ standard library flavour... ./configure: line 11487: /nix/store/r7r10qvsqlnvbzjkjinvscjlahqbxifl-gcc-wrapper-11.3.0/bin/cxx: No such file or directory
I think 'cxx' is not provided by stdenv.
Very confusingly, the `isPowerPC` predicate in
`lib/systems/inspect.nix` does *not* match `powerpc64le`!
This is because `isPowerPC` is defined as
isPowerPC = { cpu = cpuTypes.powerpc; };
Where `cpuTypes.powerpc` is:
{ bits = 32; significantByte = bigEndian; family = "power"; };
This means that the `isPowerPC` predicate actually only matches the
subset of machines marketed under this name which happen to be 32-bit
and running in big-endian mode which is equivalent to:
with stdenv.hostPlatform; isPower && isBigEndian && is32bit
This seems like a sharp edge that people could easily cut themselves
on. In fact, that has already happened: in
`linux/kernel/common-config.nix` there is a test which will always
fail:
(stdenv.hostPlatform.isPowerPC && stdenv.hostPlatform.is64bit)
A more subtle case of the strict isPowerPC being used instead of the
moreg general isPower accidentally are the GHC expressions:
Update pkgs/development/compilers/ghc/8.10.7.nix
Update pkgs/development/compilers/ghc/8.8.4.nix
Update pkgs/development/compilers/ghc/9.2.2.nix
Update pkgs/development/compilers/ghc/9.0.2.nix
Update pkgs/development/compilers/ghc/head.nix
Since the remaining legitimate use sites of isPowerPC are so few, remove
the isPowerPC predicate completely. The alternative expression above is
noted in the release notes as an alternative.
Co-authored-by: sternenseemann <sternenseemann@systemli.org>
At some point in hadrian's installation Makefile used for e.g. alpine
bindists 'xxx' is used as a placeholder for three spaces and later
substituted back. This breaks if the store path itself contains 'xxx'.
We adapt an upstream patch which will be part of 9.4 (and presumably
9.0.3 and 9.2.3) as a workaround for this issue for 8.10.2 and 8.10.7
which are the binary GHCs in nixpkgs affected.
The issue we are working around is concerned with the lack of dead
code elimination in the aarch64-darwin LLVM codegen backend. Thus
the relevant condition is the target platform, not the host
platform as checked previously.
The condition used in the past to detect iOS was "is this
aarch64-darwin"? Since we have aarch64-darwin devices running macOS
nowadays which do allow large address space, let's use the more accurate
flag.
enableDwarf requires elfutils on the host, which doesn't support darwin.
Instead of hardcoded isDarwin/isWindows, switch to self-documenting
availableOn conditional for elfutils.
Fixes ghcHEAD cross-compilation when build = host = darwin,
target = linux.
Co-authored-by: sternenseemann <sternenseemann@systemli.org>
This is no substantial change, as we already assert that the
build->target and host->target LLVM are the same, but this brings the
terminology in the file in a more consistent order, since we use
build->target for CC/CXX and bintools already.
In fact we should be passing build->target to configure always,
host->target would come into play when updating GHC's settings file
after installing.
On aarch64-darwin we have additional wrappers for install_name_tool and
strip to deal with codesigning requirements (e. g. updating the
signature / checksum after changing a binary). These wrappers don't
exist on x86_64-darwin which is why the unwrapped versions were used
before, causing the kernel to kill (some) executables produced by GHC.
CC, CXX, LD, AR, …, LLC, OPT and CLANG will be invoked by GHC's build
system at build time in the build->target role. However, since we are
passing absolute paths, they will get saved in GHC's settings file and
later invoked at runtime, when they should be host->target. This means
that the build->target and host->target tools need to be the same for
our built GHC to work properly which is what we guard using these new
asserts.
Being able to drop these asserts would be a step towards cross-compiling
GHC (as opposed to building a GHC cross-compiler which still works).
