Rahul Butani 647ac0a0d8 libxcrypt: fix linking with lld 17+ ... --- As detailed in #299849, `pkgsLLVM.libxcrypt` fails to build. At nixpkgs rev `3b61d595b17f2e3410bcc79c473060e098829eb8`: ```console $ nix-instantiate . -A libxcrypt # gcc /nix/store/finy8awd01kgg8p832m7mnm49qhm2225-libxcrypt-4.4.36.drv $ nix-instantiate . -A pkgsLLVM.libxcrypt # LLVM 17 /nix/store/wybil0v9j55681w8vwz5751j6kfaccc9-libxcrypt-x86_64-unknown-linux-gnu-4.4.36.drv ``` This is the error we get when building the latter: ```console libtool: link: x86_64-unknown-linux-gnu-clang -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ lib/.libs/libcrypt_la-alg-des.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-core.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-hmac.o \ lib/.libs/libcrypt_la-alg-hmac-sha1.o \ lib/.libs/libcrypt_la-alg-md4.o \ lib/.libs/libcrypt_la-alg-md5.o \ lib/.libs/libcrypt_la-alg-sha1.o \ lib/.libs/libcrypt_la-alg-sha256.o \ lib/.libs/libcrypt_la-alg-sha512.o \ lib/.libs/libcrypt_la-alg-yescrypt-common.o \ lib/.libs/libcrypt_la-alg-yescrypt-opt.o \ lib/.libs/libcrypt_la-crypt-bcrypt.o \ lib/.libs/libcrypt_la-crypt-des.o \ lib/.libs/libcrypt_la-crypt-gensalt-static.o \ lib/.libs/libcrypt_la-crypt-gost-yescrypt.o \ lib/.libs/libcrypt_la-crypt-md5.o \ lib/.libs/libcrypt_la-crypt-nthash.o \ lib/.libs/libcrypt_la-crypt-pbkdf1-sha1.o \ lib/.libs/libcrypt_la-crypt-scrypt.o \ lib/.libs/libcrypt_la-crypt-sha256.o \ lib/.libs/libcrypt_la-crypt-sha512.o \ lib/.libs/libcrypt_la-crypt-static.o \ lib/.libs/libcrypt_la-crypt-sunmd5.o \ lib/.libs/libcrypt_la-crypt-yescrypt.o \ lib/.libs/libcrypt_la-crypt.o \ lib/.libs/libcrypt_la-util-base64.o \ lib/.libs/libcrypt_la-util-gensalt-sha.o \ lib/.libs/libcrypt_la-util-get-random-bytes.o \ lib/.libs/libcrypt_la-util-make-failure-token.o \ lib/.libs/libcrypt_la-util-xbzero.o \ lib/.libs/libcrypt_la-util-xstrcpy.o \ -g -O2 \ -Wl,--version-script -Wl,./libcrypt.map \ -Wl,-z -Wl,defs \ -Wl,-z -Wl,text \ -Wl,-z -Wl,relro \ -Wl,-z -Wl,now \ -Wl,-soname -Wl,libcrypt.so.2 \ -o .libs/libcrypt.so.2.0.0 x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'crypt_gensalt_r' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt_r' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_r' failed: symbol not defined ``` Comparing the gcc build and the LLVM build: Here's the link invocation for `libcrypt.so.2.0.0` w/gcc: ```console libtool: link: gcc -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ lib/.libs/libcrypt_la-alg-des.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-core.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-hmac.o \ lib/.libs/libcrypt_la-alg-hmac-sha1.o \ lib/.libs/libcrypt_la-alg-md4.o \ lib/.libs/libcrypt_la-alg-md5.o \ lib/.libs/libcrypt_la-alg-sha1.o \ lib/.libs/libcrypt_la-alg-sha256.o \ lib/.libs/libcrypt_la-alg-sha512.o \ lib/.libs/libcrypt_la-alg-yescrypt-common.o \ lib/.libs/libcrypt_la-alg-yescrypt-opt.o \ lib/.libs/libcrypt_la-crypt-bcrypt.o \ lib/.libs/libcrypt_la-crypt-des.o \ lib/.libs/libcrypt_la-crypt-gensalt-static.o \ lib/.libs/libcrypt_la-crypt-gost-yescrypt.o \ lib/.libs/libcrypt_la-crypt-md5.o \ lib/.libs/libcrypt_la-crypt-nthash.o \ lib/.libs/libcrypt_la-crypt-pbkdf1-sha1.o \ lib/.libs/libcrypt_la-crypt-scrypt.o \ lib/.libs/libcrypt_la-crypt-sha256.o \ lib/.libs/libcrypt_la-crypt-sha512.o \ lib/.libs/libcrypt_la-crypt-static.o \ lib/.libs/libcrypt_la-crypt-sunmd5.o \ lib/.libs/libcrypt_la-crypt-yescrypt.o \ lib/.libs/libcrypt_la-crypt.o \ lib/.libs/libcrypt_la-util-base64.o \ lib/.libs/libcrypt_la-util-gensalt-sha.o \ lib/.libs/libcrypt_la-util-get-random-bytes.o \ lib/.libs/libcrypt_la-util-make-failure-token.o \ lib/.libs/libcrypt_la-util-xbzero.o \ lib/.libs/libcrypt_la-util-xstrcpy.o \ -g -O2 \ -Wl,--version-script -Wl,./libcrypt.map \ -Wl,-z -Wl,defs \ -Wl,-z -Wl,text \ -Wl,-z -Wl,relro \ -Wl,-z -Wl,now \ -Wl,-soname -Wl,libcrypt.so.2 \ -o .libs/libcrypt.so.2.0.0 ``` (i.e. they're identical, modulo the compiler name: ```diff *** gcc-link-invocation 2024-05-07 07:17:46.034713173 -0700 --- clang-link-invocation 2024-05-07 07:19:38.030392378 -0700 *************** *** 1,3 **** ! libtool: link: gcc -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ --- 1,3 ---- ! libtool: link: x86_64-unknown-linux-gnu-clang -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ ``` ) `libcrypt.map` is also identical for the two builds: ```ld /* Generated from libcrypt.map.in by gen-libcrypt-map. DO NOT EDIT. */ XCRYPT_2.0 { global: crypt; crypt_gensalt; crypt_gensalt_r; crypt_gensalt_ra; crypt_gensalt_rn; crypt_r; crypt_ra; crypt_rn; xcrypt; xcrypt_gensalt; xcrypt_gensalt_r; xcrypt_r; local: *; }; XCRYPT_4.3 { global: crypt_checksalt; } XCRYPT_2.0; XCRYPT_4.4 { global: crypt_preferred_method; } XCRYPT_4.3; ``` Comparing the defined exported symbols in the `.o` files linked in across the two builds: ```bash objs=( alg-des-tables.o alg-des.o alg-gost3411-2012-core.o alg-gost3411-2012-hmac.o alg-hmac-sha1.o alg-md4.o alg-md5.o alg-sha1.o alg-sha256.o alg-sha512.o alg-yescrypt-common.o alg-yescrypt-opt.o crypt-bcrypt.o crypt-des.o crypt-gensalt-static.o crypt-gost-yescrypt.o crypt-md5.o crypt-nthash.o crypt-pbkdf1-sha1.o crypt-scrypt.o crypt-sha256.o crypt-sha512.o crypt-static.o crypt-sunmd5.o crypt-yescrypt.o crypt.o util-base64.o util-gensalt-sha.o util-get-random-bytes.o util-make-failure-token.o util-xbzero.o util-xstrcpy.o ) for build_dir in gcc llvm; do for obj in ${objs[@]}; do o=$build_dir/lib/.libs/libcrypt_la-$obj nm -C --just-symbols --defined-only --extern-only $o done > ${build_dir}_syms.txt done diff {gcc,llvm}_syms.txt ``` ... yields no differences. Here's the full list: ```txt _crypt_GOST34112012_Cleanup _crypt_GOST34112012_Final _crypt_GOST34112012_Init _crypt_GOST34112012_Update _crypt_gost_hash256 _crypt_gost_hmac256 _crypt_HMAC_SHA256_Buf _crypt_HMAC_SHA256_Final _crypt_HMAC_SHA256_Init _crypt_HMAC_SHA256_Update _crypt_PBKDF2_SHA256 _crypt_SHA256_Buf _crypt_SHA256_Final _crypt_SHA256_Init _crypt_SHA256_Update _crypt_SHA512_Buf _crypt_SHA512_Final _crypt_SHA512_Init _crypt_SHA512_Update _crypt_crypto_scrypt _crypt_yescrypt _crypt_yescrypt_decode64 _crypt_yescrypt_encode64 _crypt_yescrypt_encode_params _crypt_yescrypt_encode_params_r _crypt_yescrypt_r _crypt_yescrypt_reencrypt _crypt_yescrypt_digest_shared _crypt_yescrypt_free_local _crypt_yescrypt_free_shared _crypt_yescrypt_init_local _crypt_yescrypt_init_shared _crypt_yescrypt_kdf _crypt_crypt_bcrypt_a_rn _crypt_crypt_bcrypt_rn _crypt_crypt_bcrypt_y_rn _crypt_gensalt_bcrypt_a_rn _crypt_gensalt_bcrypt_rn _crypt_gensalt_bcrypt_y_rn _crypt_crypt_gensalt crypt_gensalt@@XCRYPT_2.0 _crypt_crypt_gost_yescrypt_rn _crypt_gensalt_gost_yescrypt_rn _crypt_crypt_scrypt_rn _crypt_gensalt_scrypt_rn _crypt_crypt_sha512crypt_rn _crypt_gensalt_sha512crypt_rn _crypt_crypt crypt@@XCRYPT_2.0 _crypt_crypt_yescrypt_rn _crypt_gensalt_yescrypt_rn crypt_checksalt@@XCRYPT_4.3 _crypt_crypt_checksalt _crypt_crypt_gensalt_ra _crypt_crypt_gensalt_rn _crypt_crypt_preferred_method _crypt_crypt_r _crypt_crypt_ra _crypt_crypt_rn crypt_gensalt_ra@@XCRYPT_2.0 crypt_gensalt_rn@@XCRYPT_2.0 crypt_preferred_method@@XCRYPT_4.4 crypt_ra@@XCRYPT_2.0 crypt_rn@@XCRYPT_2.0 crypt_r@@XCRYPT_2.0 _crypt_ascii64 _crypt_gensalt_sha_rn _crypt_get_random_bytes _crypt_make_failure_token _crypt_strcpy_or_abort ``` All of the symbols `lld` was complaining about actually are missing: ``` crypt_gensalt_r xcrypt xcrypt_gensalt xcrypt_gensalt_r xcrypt_r ``` --- Poking around a bit reveals that this is really a regression caused by the [`llvmPackages` 16 -> 17 bump on Linux](https://github.com/NixOS/nixpkgs/pull/285333); digging a little further reveals that this is ultimately caused by a change in `lld`: ```console // in the nix repl > np = import ./. {} // use gcc w/lld 16: > okay = np.libxcrypt.override { stdenv = np.stdenv.override (o: { allowedRequisites = null; cc = o.cc.override { bintools = np.llvmPackages_16.bintools; }; }); } > okay «derivation /nix/store/mdpaggjkjiw7v2qzw8g4xaf9vfw3z28d-libxcrypt-4.4.36.drv» > :b okay # builds fine, modulo some silliness with flags // use gcc w/lld *17*: > bad = np.libxcrypt.override { stdenv = np.stdenv.override (o: { allowedRequisites = null; cc = o.cc.override { bintools = np.llvmPackages_17.bintools; }; }); } > bad «derivation /nix/store/66bjyyapykl6mbmai7qbn7vwm762bdsc-libxcrypt-4.4.36.drv» > :b bad # yields the version script linker error ``` Note that with the `gcc` + `lld` 16 build above we get the following output as part of the build: ``` ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'crypt_gensalt_r' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt_r' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_r' failed: symbol not defined ``` This is because in 16 `--no-undefined-version` is enabled by default: - [commit](241dbd3105) - [phabricator](https://reviews.llvm.org/D135402) Errors were ultimately demoted to warnings for the LLVM 16 release: - [issue](https://github.com/llvm/llvm-project/issues/61208) - [PR](https://github.com/llvm/llvm-project-release-prs/pull/347) - [commit](c1949c6a31) As a workaround, this PR adds in `--undefined-version` as a linker flag when building `libxcrypt` with `lld` 17 and newer. Ultimately, the "right" solution is probably for upstream to update `libcrypt.map` to not reference these symbols when they're not present in `libcrypt.so`; see: https://github.com/besser82/libxcrypt/issues/181 --- Closes #299849.		2024-05-07 23:48:00 -07:00
..
check_passthru_matches.py
default.nix