nordic-dev.net/rust
nordic-dev.net/rust
2
0
Fork 0
mirror of https://github.com/rust-lang/rust.git synced 2025-02-03 02:23:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Auto merge of #58575 - mati865:musl_toolchain, r=alexcrichton

Browse Source 
Musl host toolchain

Based on https://github.com/rust-lang/rust/pull/55163 and https://github.com/rust-lang/rust/pull/57359
Depends on https://github.com/rust-lang/rust/pull/55566

CC https://github.com/rust-lang/rust/issues/57439

### How it works

Tested compiler made by `dist` on glibc and musl based distributions and verified binaries it produces:
* Ubuntu (glibc) - installed it as a target for host toolchain and observed no regressions for static (default) linking, dynamic linking apparently requires musl build libgcc so I didn't test it.
* Alpine (musl) - installed as the host toolchain, by default it links statically (executables are portable and work on glibc distributions) but with `-C target-feature=-crt-static` Rust flag it links dynamically (executables require musl built libraries).

### What's debatable

It should be decided whether this toolchain should link dynamically or statically when using it on musl distribution. I believe the distributions would prefer dynamic linking but it'd be misleading because `$ARCH-unknown-linux-musl` target links statically on the other hosts.
Another problem is using `RUSTFLAGS='-C target-feature=-crt-static'` for dynamic builds which is really uncomfortable.

To address both issues I suggest leaving `$ARCH-unknown-linux-musl` static for both host and cross target and introducing "alias triple" `$ARCH-unknown-linux-dynmusl`. It'd be the same as `$ARCH-unknown-linux-musl` (and use the same libraries to avoid duplication) but it'd link dynamically.

<del>
### Why it's still WIP (help wanted)

I'm having a hard time getting all tests to pass and I'd appreciate help.

Non-verbose error:
<details>

```
Testing proc_macro stage1 (x86_64-unknown-linux-musl -> x86_64-unknown-linux-musl)
   Compiling proc_macro v0.0.0 (/checkout/src/libproc_macro)
error[E0463]: can't find crate for `std`

error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
[RUSTC-TIMING] proc_macro test:true 0.529
[RUSTC-TIMING] proc_macro test:false 0.530
error: Could not compile `proc_macro`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `proc_macro`.

To learn more, run the command again with --verbose.

command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-musl" "-j" "16" "--release" "--locked" "--manifest-path" "/checkout/src/libtest/Cargo.toml" "-p" "proc_macro" "--"
expected success, got: exit code: 101

failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --host x86_64-unknown-linux-musl --target x86_64-unknown-linux-musl
```
</details>

Verbose error:
<details>

```
Testing proc_macro stage1 (x86_64-unknown-linux-musl -> x86_64-unknown-linux-musl)
   Compiling proc_macro v0.0.0 (/checkout/src/libproc_macro)
     Running `/checkout/obj/build/bootstrap/debug/rustc --edition=2018 --crate-name proc_macro src/libproc_macro/lib.rs --color never --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=09ddd3ecc930ab63 -C extra-filename=-09ddd3ecc930ab63 --out-dir /checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps --target x86_64-unknown-linux-musl -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/release/deps -C target-feature=-crt-static`
     Running `/checkout/obj/build/bootstrap/debug/rustc --edition=2018 --crate-name proc_macro src/libproc_macro/lib.rs --color never --emit=dep-info,link -C opt-level=2 --test -C metadata=a564d363930469c8 -C extra-filename=-a564d363930469c8 --out-dir /checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps --target x86_64-unknown-linux-musl -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/release/deps -C target-feature=-crt-static`
error[E0463]: can't find crate for `std`
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
[RUSTC-TIMING] proc_macro test:false 0.248
error: Could not compile `proc_macro`.

Caused by:
  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --edition=2018 --crate-name proc_macro src/libproc_macro/lib.rs --color never --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=09ddd3ecc930ab63 -C extra-filename=-09ddd3ecc930ab63 --out-dir /checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps --target x86_64-unknown-linux-musl -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/release/deps -C target-feature=-crt-static` (exit code: 1)
warning: build failed, waiting for other jobs to finish...
[RUSTC-TIMING] proc_macro test:true 0.248
error: Could not compile `proc_macro`.

Caused by:
  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --edition=2018 --crate-name proc_macro src/libproc_macro/lib.rs --color never --emit=dep-info,link -C opt-level=2 --test -C metadata=a564d363930469c8 -C extra-filename=-a564d363930469c8 --out-dir /checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps --target x86_64-unknown-linux-musl -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-musl/stage1-test/release/deps -C target-feature=-crt-static` (exit code: 1)

command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-musl" "-j" "16" "--release" "--locked" "--manifest-path" "/checkout/src/libtest/Cargo.toml" "--verbose" "-p" "proc_macro" "--"
expected success, got: exit code: 101

failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --host x86_64-unknown-linux-musl --target x86_64-unknown-linux-musl
```

</details>

Whole tests non-verbose output: [rust-tests.log](https://github.com/rust-lang/rust/files/2879945/rust-tests.log)

I think the error is because build system (correctly?) tries to use `obj/build/x86_64-unknown-linux-musl/stage1-test/x86_64-unknown-linux-musl/release/deps` which is empty but `obj/build/x86_64-unknown-linux-gnu/stage1-test/x86_64-unknown-linux-musl/release/deps` contains required libs.
</del>
This commit is contained in:
bors 2019-03-15 11:00:13 +00:00
commit 16e7e05e95
3 changed files with 99 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
src/ci/docker/dist-x86_64-musl/Dockerfile
View File

@ -4,6 +4,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
g++ \
make \
file \
wget \
curl \
ca-certificates \
python2.7 \
@ -18,19 +19,17 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
WORKDIR /build/
COPY scripts/musl.sh /build/
COPY scripts/musl-toolchain.sh /build/
# We need to mitigate rust-lang/rust#34978 when compiling musl itself as well
RUN CC=gcc \
CFLAGS="-Wa,-mrelax-relocations=no" \
CXX=g++ \
RUN CFLAGS="-Wa,-mrelax-relocations=no" \
CXXFLAGS="-Wa,-mrelax-relocations=no" \
bash musl.sh x86_64 && rm -rf /build
bash musl-toolchain.sh x86_64 && rm -rf build
COPY scripts/sccache.sh /scripts/
RUN sh /scripts/sccache.sh
ENV RUST_CONFIGURE_ARGS \
--musl-root-x86_64=/musl-x86_64 \
--musl-root-x86_64=/usr/local/x86_64-linux-musl \
--enable-extended \
--disable-docs
@ -41,6 +40,12 @@ ENV RUST_CONFIGURE_ARGS \
# See: https://github.com/rust-lang/rust/issues/34978
ENV CFLAGS_x86_64_unknown_linux_musl=-Wa,-mrelax-relocations=no
ENV SCRIPT \
python2.7 ../x.py test --target x86_64-unknown-linux-musl && \
python2.7 ../x.py dist --target x86_64-unknown-linux-musl
ENV HOSTS=x86_64-unknown-linux-musl \
CC_x86_64_unknown_linux_musl=x86_64-linux-musl-gcc \
CXX_x86_64_unknown_linux_musl=x86_64-linux-musl-g++
# Musl defaults to static libs but we need them to be dynamic for host toolchain.
# The toolchain will produce static libs by default.
ENV RUSTFLAGS="-C target-feature=-crt-static"
ENV SCRIPT python2.7 ../x.py dist --host $HOSTS --target $HOSTS

70
src/ci/docker/scripts/musl-toolchain.sh Normal file
View File

@ -0,0 +1,70 @@
# This script runs `musl-cross-make` to prepare C toolchain (Binutils, GCC, musl itself)
# and builds static libunwind that we distribute for static target.
#
# Versions of the toolchain components are configurable in `musl-cross-make/Makefile` and
# musl unlike GLIBC is forward compatible so upgrading it shouldn't break old distributions.
# Right now we have: Binutils 2.27, GCC 6.3.0, musl 1.1.18
set -ex
hide_output() {
set +x
on_err="
echo ERROR: An error was encountered with the build.
cat /tmp/build.log
exit 1
"
trap "$on_err" ERR
bash -c "while true; do sleep 30; echo \$(date) - building ...; done" &
PING_LOOP_PID=$!
$@ &> /tmp/build.log
trap - ERR
kill $PING_LOOP_PID
rm /tmp/build.log
set -x
}
ARCH=$1
TARGET=$ARCH-linux-musl
OUTPUT=/usr/local
shift
git clone https://github.com/richfelker/musl-cross-make -b v0.9.7
cd musl-cross-make
hide_output make -j$(nproc) TARGET=$TARGET
hide_output make install TARGET=$TARGET OUTPUT=$OUTPUT
cd -
# Install musl library to make binaries executable
ln -s $OUTPUT/$TARGET/lib/libc.so /lib/ld-musl-$ARCH.so.1
echo $OUTPUT/$TARGET/lib >> /etc/ld-musl-$ARCH.path
export CC=$TARGET-gcc
export CXX=$TARGET-g++
LLVM=70
# may have been downloaded in a previous run
if [ ! -d libunwind-release_$LLVM ]; then
curl -L https://github.com/llvm-mirror/llvm/archive/release_$LLVM.tar.gz | tar xzf -
curl -L https://github.com/llvm-mirror/libunwind/archive/release_$LLVM.tar.gz | tar xzf -
fi
# fixme(mati865): Replace it with https://github.com/rust-lang/rust/pull/59089
mkdir libunwind-build
cd libunwind-build
cmake ../libunwind-release_$LLVM \
-DLLVM_PATH=/build/llvm-release_$LLVM \
-DLIBUNWIND_ENABLE_SHARED=0 \
-DCMAKE_C_COMPILER=$CC \
-DCMAKE_CXX_COMPILER=$CXX \
-DCMAKE_C_FLAGS="$CFLAGS" \
-DCMAKE_CXX_FLAGS="$CXXFLAGS"
hide_output make -j$(nproc)
cp lib/libunwind.a $OUTPUT/$TARGET/lib
cd - && rm -rf libunwind-build

17
src/ci/docker/test-various/Dockerfile
View File

@ -11,7 +11,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
cmake \
sudo \
gdb \
xz-utils
xz-utils \
wget \
patch
# FIXME: build the `ptx-linker` instead.
RUN curl -sL https://github.com/denzp/rust-ptx-linker/releases/download/v0.9.0-alpha.2/rust-ptx-linker.linux64.tar.gz | \
@ -20,10 +22,16 @@ RUN curl -sL https://github.com/denzp/rust-ptx-linker/releases/download/v0.9.0-a
RUN curl -sL https://nodejs.org/dist/v9.2.0/node-v9.2.0-linux-x64.tar.xz | \
tar -xJ
WORKDIR /build/
COPY scripts/musl-toolchain.sh /build/
RUN bash musl-toolchain.sh x86_64 && rm -rf build
WORKDIR /
COPY scripts/sccache.sh /scripts/
RUN sh /scripts/sccache.sh
ENV RUST_CONFIGURE_ARGS \
--musl-root-x86_64=/usr/local/x86_64-linux-musl \
--set build.nodejs=/node-v9.2.0-linux-x64/bin/node \
--set rust.lld
@ -47,4 +55,9 @@ ENV NVPTX_TARGETS=nvptx64-nvidia-cuda
ENV NVPTX_SCRIPT python2.7 /checkout/x.py test --target $NVPTX_TARGETS \
src/test/run-make
ENV SCRIPT $WASM_SCRIPT && $NVPTX_SCRIPT
ENV MUSL_TARGETS=x86_64-unknown-linux-musl \
CC_x86_64_unknown_linux_musl=x86_64-linux-musl-gcc \
CXX_x86_64_unknown_linux_musl=x86_64-linux-musl-g++
ENV MUSL_SCRIPT python2.7 /checkout/x.py test --target $MUSL_TARGETS
ENV SCRIPT $WASM_SCRIPT && $NVPTX_SCRIPT && $MUSL_SCRIPT