Also begin to start work on cross compilation, though that will have to
be finished later.
The patches are based on the first version of
https://reviews.llvm.org/D99484. It's very annoying to do the
back-porting but the review has uncovered nothing super major so I'm
fine sticking with what I've got.
Beyond making the outputs work, I also strove to re-sync the packages,
as they have been drifting pointlessly apart for some time.
----
Other misc notes, highly incomplete
- lvm-config-native and llvm-config are put in `dev` because they are
tools just for build time.
- Clang no longer has an lld dep. That was introduced in
db29857eb3, but if clang needs help
finding lld when it is used we should just pass it flags / put in the
resource dir. Providing it at build time increases critical path
length for no good reason.
----
A note on `nativeCC`:
`stdenv` takes tools from the previous stage, so:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.stdenv.cc`: `(?0, ?1, x)`
while:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.targetPackages`: `(x, x, ?2)`
3. `pkgsBuildBuild.targetPackages.stdenv.cc`: `(?1, x, x)`
Use the same JDK for building bazel and for its runtime.
Effectively, the former `toolchain_hostjdk8` java toolchain has been deprecated
and should no longer be used (in newer bazel)[1]:
```
default_java_toolchain(
name = "toolchain_hostjdk8",
...
)
```
[1]: 4fc4868065/tools/jdk/BUILD.tools (L384-L387)
* use default stdenv (clang 7)
* add no-arc.patch to make the xcode_locate tool compile without libarc-lite
* remove the `-mmacosx-version-min=10.9` flag from the bootstrap compile script
- bump protobuf to 3.13.0
- use recent rules_proto
- use recent bazel-skylib
- remove use of native.bind
Hopefully this is backward-compatible with bazel_3 and below
Bazel 4 is going to be a long term support release.
Latest version in NixPkgs so far was 3.3.1
There's a need for more recent version
https://github.com/NixOS/nixpkgs/issues/97497
All versions from 3.5.0 to 3.7.1 had some reproducibility issues
as noted in issue above, but there also seems to be
a working PR for 3.7.1 now at
https://github.com/NixOS/nixpkgs/pull/105439
Notable changes from bazel_3 setup:
- put python to default bash path
For autodetecting python toolchain
with strict action_env on and without this change
bazel would fail to autodetect host python.
There are some repos that define hermetic python
toolchains, but they aren't easy to use yet. Also
telling python paths to bazel isn't a 1-liner it
seems:
- action_env=PATH would affect cache
- declaring toolchain via BUILD&WORKSPACE files
is not per-user but more like per-repo and
affects cache too
Using python from nixpkgs shouldn't be too bad
in the lack of simpler hermetic python toolchain
options
- bazel_4.updater is bazel on `bazel query` to support
new constructs in WORKSPACE (load of vars, transitive
load etc). This is more robust but requires bazel
to run the updater, using bazel_3 for now. This is
only needed to bump package version, doesn't introduce
bazel_4 build dependency on bazel_3
https://blog.bazel.build/2020/11/10/bazel-4.0-announce.htmlhttps://blog.bazel.build/2020/11/10/long-term-support-release.htmlhttps://github.com/bazelbuild/bazel/issues/12455https://github.com/bazelbuild/bazel/releases/tag/4.0.0https://blog.bazel.build/2021/01/19/bazel-4-0.html
* use default stdenv (clang 7)
* add no-arc.patch to make the xcode_locate tool compile without libarc-lite
* remove the `-mmacosx-version-min=10.9` flag from the bootstrap compile script
Use the same JDK for building bazel and for its runtime.
Effectively, the former `toolchain_hostjdk8` java toolchain has been deprecated
and should no longer be used (in newer bazel)[1]:
```
# Deprecated, do not use.
# It will be removed after migration to Java toolchain resolution.
default_java_toolchain(
name = "toolchain_hostjdk8",
...
)
```
[1]: 4fc4868065/tools/jdk/BUILD.tools (L384-L387)
These are fixes for problems I ran into with:
- `bazel test //example:cpp-test`
This needed `build --host_javabase='@local_jdk//:jdk'`
- `bazel query 'deps(//example:cpp-test)'`
This needed the same flags as `build`.
Is it contentious to (partially?) configure the default java toolchain? I don't see it as much different than providing the bazel server's java.
It would continue to be configurable/overridable by overriding the flags.
---
And a random notes from this escapade, but https://github.com/bazelbuild/bazel/blob/master/WORKSPACE#L144-L308 looks a little different from https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel/bazel_3/src-deps.json so one of them is probably wrong :)
Limit the resources Bazel is allowed to use during the build to 1/2 the
available RAM and 3/4 the available CPU cores. This should help avoid
overwhelming the build machine.
Preserving existing behavior: the bash completion was not executable,
the zsh completion was; according to lukegb the fish completion does
not have to be executable.
flat hashes can be substituted through hashed-mirrors, while recursive
hashes can’t. This is especially important for Bazel since the bazel
fetch dependencies can come from multiple different methods (git,
http, ftp, etc.). To do this, we create tar archives from the
output/external directory, which is then extracted to build. All of
the Bazel hashes are all updated.
Using `bazel_self` for self-references makes managing bazel versions
easier: their less risk of changing defaults or copy pasted code for no
versions leading to incorrect self-references.
From Bazel 2.0.0 onwards, Bazel looks for a binary named
`bazel-${version}-${os_arch}` if the project root contains a
`.bazelversion` file or the USE_BAZEL_VERSION environment
variable is set.
This change ensures we output a binary with the correct name
for the current version and OS/arch combination.
- Bazel strict action env set a default PATH to
`/bin/:/usr/bin:/usr/local/bin`. This was previously changed to disable
this behavior to improve hermeticity. However the previous change was
only removing `/bin:/usr/bin`, keeping `/usr/local/bin`, this commit
also remove this entry.
- Upgraded dependencies
- dependencies script upgraded to take into account new WORKSPACE
rules
- Tests now depends on the `distdir`
Runtime bazel now also depends on the `distdir` setting which appears
in the global configuration file. This increases the bazel closure
size by 85 MO for stuffs which can normally be downloaded at runtime
by bazel. However, any invocation of `buildBazelPackage` (such as in
`bazel-watcher`) may fail in nix sandbox if theses files are not
available at runtime.
If this overhead is too important, we may later evolve to a finer
grained solution, where buildBazelPackage declares the list of
necessary dependencies.
