rust/tests/run-make/long-linker-command-lines-cmd-exe/foo.rs

102 lines
3.3 KiB
Rust
Raw Normal View History

rustc: Lower link args to `@`-files on Windows more When spawning a linker rustc has historically been known to blow OS limits for the command line being too large, notably on Windows. This is especially true of incremental compilation where there can be dozens of object files per compilation. The compiler currently has logic for detecting a failure to spawn and instead passing arguments via a file instead, but this failure detection only triggers if a process actually fails to spawn. Unfortunately on Windows we've got something else to worry about which is `cmd.exe`. The compiler may be running a linker through `cmd.exe` where `cmd.exe` has a limit of 8192 on the command line vs 32k on `CreateProcess`. Moreso rustc actually succeeds in spawning `cmd.exe` today, it's just that after it's running `cmd.exe` fails to spawn its child, which rustc doesn't currently detect. Consequently this commit updates the logic for the spawning the linker on Windows to instead have a heuristic to see if we need to pass arguments via a file. This heuristic is an overly pessimistic and "inaccurate" calculation which just calls `len` on a bunch of `OsString` instances (where `len` is not precisely the length in u16 elements). This number, when exceeding the 6k threshold, will force rustc to always pass arguments through a file. This strategy should avoid us trying to parse the output on Windows of the linker to see if it successfully spawned yet failed to actually sub-spawn the linker. We may just be passing arguments through files a little more commonly now... The motivation for this commit was a recent bug in Gecko [1] when beta testing, notably when incremental compilation was enabled it blew out the limit on `cmd.exe`. This commit will also fix #46999 as well though as emscripten uses a bat script as well (and we're blowing the limit there). [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1430886 Closes #46999
2018-01-16 23:30:57 +00:00
// Like the `long-linker-command-lines` test this test attempts to blow
// a command line limit for running the linker. Unlike that test, however,
// this test is testing `cmd.exe` specifically rather than the OS.
//
// Unfortunately `cmd.exe` has a 8192 limit which is relatively small
// in the grand scheme of things and anyone sripting rustc's linker
// is probably using a `*.bat` script and is likely to hit this limit.
//
// This test uses a `foo.bat` script as the linker which just simply
// delegates back to this program. The compiler should use a lower
// limit for arguments before passing everything via `@`, which
// means that everything should still succeed here.
use std::env;
use std::fs::{self, File};
use std::io::{BufWriter, Write, Read};
use std::path::PathBuf;
use std::process::Command;
fn main() {
if !cfg!(windows) {
return
}
let tmpdir = PathBuf::from(env::var_os("OUT_DIR").unwrap());
let ok = tmpdir.join("ok");
let not_ok = tmpdir.join("not_ok");
if env::var("YOU_ARE_A_LINKER").is_ok() {
match env::args_os().find(|a| a.to_string_lossy().contains("@")) {
Some(file) => {
let file = file.to_str().unwrap();
fs::copy(&file[1..], &ok).unwrap();
}
rustc: Lower link args to `@`-files on Windows more When spawning a linker rustc has historically been known to blow OS limits for the command line being too large, notably on Windows. This is especially true of incremental compilation where there can be dozens of object files per compilation. The compiler currently has logic for detecting a failure to spawn and instead passing arguments via a file instead, but this failure detection only triggers if a process actually fails to spawn. Unfortunately on Windows we've got something else to worry about which is `cmd.exe`. The compiler may be running a linker through `cmd.exe` where `cmd.exe` has a limit of 8192 on the command line vs 32k on `CreateProcess`. Moreso rustc actually succeeds in spawning `cmd.exe` today, it's just that after it's running `cmd.exe` fails to spawn its child, which rustc doesn't currently detect. Consequently this commit updates the logic for the spawning the linker on Windows to instead have a heuristic to see if we need to pass arguments via a file. This heuristic is an overly pessimistic and "inaccurate" calculation which just calls `len` on a bunch of `OsString` instances (where `len` is not precisely the length in u16 elements). This number, when exceeding the 6k threshold, will force rustc to always pass arguments through a file. This strategy should avoid us trying to parse the output on Windows of the linker to see if it successfully spawned yet failed to actually sub-spawn the linker. We may just be passing arguments through files a little more commonly now... The motivation for this commit was a recent bug in Gecko [1] when beta testing, notably when incremental compilation was enabled it blew out the limit on `cmd.exe`. This commit will also fix #46999 as well though as emscripten uses a bat script as well (and we're blowing the limit there). [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1430886 Closes #46999
2018-01-16 23:30:57 +00:00
None => { File::create(&not_ok).unwrap(); }
}
return
}
let rustc = env::var_os("RUSTC").unwrap_or("rustc".into());
let me = env::current_exe().unwrap();
let bat = me.parent()
.unwrap()
.join("foo.bat");
let bat_linker = format!("linker={}", bat.display());
for i in (1..).map(|i| i * 10) {
println!("attempt: {}", i);
let file = tmpdir.join("bar.rs");
let mut f = BufWriter::new(File::create(&file).unwrap());
let mut lib_name = String::new();
for _ in 0..i {
lib_name.push_str("foo");
}
for j in 0..i {
writeln!(f, "#[link(name = \"{}{}\")]", lib_name, j).unwrap();
}
writeln!(f, "extern {{}}\nfn main() {{}}").unwrap();
f.into_inner().unwrap();
drop(fs::remove_file(&ok));
drop(fs::remove_file(&not_ok));
let status = Command::new(&rustc)
.arg(&file)
.arg("-C").arg(&bat_linker)
.arg("--out-dir").arg(&tmpdir)
.env("YOU_ARE_A_LINKER", "1")
.env("MY_LINKER", &me)
.status()
.unwrap();
if !status.success() {
panic!("rustc didn't succeed: {}", status);
}
if !ok.exists() {
assert!(not_ok.exists());
continue
}
let mut contents = Vec::new();
File::open(&ok).unwrap().read_to_end(&mut contents).unwrap();
rustc: Lower link args to `@`-files on Windows more When spawning a linker rustc has historically been known to blow OS limits for the command line being too large, notably on Windows. This is especially true of incremental compilation where there can be dozens of object files per compilation. The compiler currently has logic for detecting a failure to spawn and instead passing arguments via a file instead, but this failure detection only triggers if a process actually fails to spawn. Unfortunately on Windows we've got something else to worry about which is `cmd.exe`. The compiler may be running a linker through `cmd.exe` where `cmd.exe` has a limit of 8192 on the command line vs 32k on `CreateProcess`. Moreso rustc actually succeeds in spawning `cmd.exe` today, it's just that after it's running `cmd.exe` fails to spawn its child, which rustc doesn't currently detect. Consequently this commit updates the logic for the spawning the linker on Windows to instead have a heuristic to see if we need to pass arguments via a file. This heuristic is an overly pessimistic and "inaccurate" calculation which just calls `len` on a bunch of `OsString` instances (where `len` is not precisely the length in u16 elements). This number, when exceeding the 6k threshold, will force rustc to always pass arguments through a file. This strategy should avoid us trying to parse the output on Windows of the linker to see if it successfully spawned yet failed to actually sub-spawn the linker. We may just be passing arguments through files a little more commonly now... The motivation for this commit was a recent bug in Gecko [1] when beta testing, notably when incremental compilation was enabled it blew out the limit on `cmd.exe`. This commit will also fix #46999 as well though as emscripten uses a bat script as well (and we're blowing the limit there). [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1430886 Closes #46999
2018-01-16 23:30:57 +00:00
for j in 0..i {
let exp = format!("{}{}", lib_name, j);
let exp = if cfg!(target_env = "msvc") {
let mut out = Vec::with_capacity(exp.len() * 2);
for c in exp.encode_utf16() {
// encode in little endian
out.push(c as u8);
out.push((c >> 8) as u8);
}
out
} else {
exp.into_bytes()
};
assert!(contents.windows(exp.len()).any(|w| w == &exp[..]));
rustc: Lower link args to `@`-files on Windows more When spawning a linker rustc has historically been known to blow OS limits for the command line being too large, notably on Windows. This is especially true of incremental compilation where there can be dozens of object files per compilation. The compiler currently has logic for detecting a failure to spawn and instead passing arguments via a file instead, but this failure detection only triggers if a process actually fails to spawn. Unfortunately on Windows we've got something else to worry about which is `cmd.exe`. The compiler may be running a linker through `cmd.exe` where `cmd.exe` has a limit of 8192 on the command line vs 32k on `CreateProcess`. Moreso rustc actually succeeds in spawning `cmd.exe` today, it's just that after it's running `cmd.exe` fails to spawn its child, which rustc doesn't currently detect. Consequently this commit updates the logic for the spawning the linker on Windows to instead have a heuristic to see if we need to pass arguments via a file. This heuristic is an overly pessimistic and "inaccurate" calculation which just calls `len` on a bunch of `OsString` instances (where `len` is not precisely the length in u16 elements). This number, when exceeding the 6k threshold, will force rustc to always pass arguments through a file. This strategy should avoid us trying to parse the output on Windows of the linker to see if it successfully spawned yet failed to actually sub-spawn the linker. We may just be passing arguments through files a little more commonly now... The motivation for this commit was a recent bug in Gecko [1] when beta testing, notably when incremental compilation was enabled it blew out the limit on `cmd.exe`. This commit will also fix #46999 as well though as emscripten uses a bat script as well (and we're blowing the limit there). [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1430886 Closes #46999
2018-01-16 23:30:57 +00:00
}
break
}
}