nixpkgs/pkgs/build-support/setup-hooks/patch-ppd-files/patch-ppd-hook.sh

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

184 lines
7.4 KiB
Bash
Raw Normal View History

patchPpdFilesHook: new setup hook for absolute executable paths PostScript Printer Description (ppd) files describe printer features and capabilities. They are usually evaluated by CUPS to convert print jobs into a format suitable for a printer. The conversion is often accomplished by commands or even short shell scripts inside the ppd files. ppd files are included in many printer driver packages. Their scripts sometimes refer to other executables; some of them are more common (like `perl`), others are more exotic (like `rastertohp`). If an executable is called with its name alone, the effects of the ppd file depend on whether the executable is in the PATH of CUPS, and on the executable's version. If an executable is called with an absolut path (like `/usr/bin/perl`), it won't work at all in NixOS. The commit at hand adds a setup hook that uses the `fixupPhase` to substitute certain executable's invocations in pdd files with absolute paths. To use it, add the hook to `nativeBuildInputs` and provide a list of executable names in `ppdFileCommands`. Each executable must be available in the package that is being built, or in `buildInputs`. The setup hook's script then looks for ppd files in `share/cups/model` and `share/ppds` in each output, and replaces executable names with their absolute paths. If ppd files need to be patched in unorthodox locations or the setup hook needs to be invoked manually for other reasons, one may leave the list `ppdFileCommands` empty to avoid automatic processing of ppd files, then call the shell function `patchPpdFileCommands` directly. Details are described in the file `patch-ppd-hook.sh`. Notes on the motivation for this setup hook: Most packages in nixpkgs that provide ppd files do not patch those ppd files at all. This is not fatal when the executables are just called with their names since the user can add packages with the executables to `services.printing.drivers`. E.g. if the user adds `pkgs.perl`, then all ppd files that invoke `perl` will work as expected. Nevertheless, to make these ppd files independent of their execution environment, command invocations should be substituted with absolut paths into the nix store. This is similar to patching shebang lines so scripts can be called independently of having the interpreter in the PATH. The hook script in this commit is meant to support new packages `foomatic-db*` which will generate several thousands of ppd files referencing a plethora of different executables. During development of these packages, I realized that it's quite hard to patch ppd files in a robust way. While binary names like `rastertokpsl` seem to be sufficiently unique to be patched with `sed`, names like `date` or `gs` are hard to patch without producing "false positives", i.e., coincidental occurences of the executable's name that do *not* refer to the executable and should not be patched at all. As this problem also affects other packages, it seems reasonable to put a robust implementation in its own setup hook so that other packages can use it without much effort. Notes on the implementation: The ppd file format is far from trivial. The basic structure are key-value pairs; keys may occur multiple times. Only a small subset of keys may contain executable names or shell scripts in their values. Some values may span multiple lines; a linebreak might even occur in the middle of a token. Some executable names also occur in other keys by accident where they must not be substituted (e.g. `gs` or `date`). It is necessary to provide the list of command names that will be patched for two reasons: ppd files often contain "tokens" that might look like commands (e.g. "file" or "host") but aren't; these would erroneously get patched. Also, looking for everything that might be a command would slow down the patching process considerably. The implementation uses `awk` to detect keys that might contain executable names; only their values are treated for substitution. This avoids most cases of "overzealous" substitutions. Since values may span multiple lines, `sed` alone (while faster than `awk`) cannot focus its substitution capabilities on relevant keys. An elaborate set of regular expressions further helps to minimize the probability of "false positives". Several tricks are employed to speed up `awk`. Notably, relevant files are identified with `grep` before `awk` is applied to those files only. Note that the script probably cannot handle fancy command names (like spaces or backslashes as part of the name). Also, there are still edge cases that the script would mistakenly skip, e.g. if a shell script contains a line break in the middle of an executable's name; although ppd files permit such constellations, I have yet to see one. ppd files may be gzipped. The setup hook accepts gzipped ppd files: It decompresses them, substitutes paths, then recompresses them. However, Nix cannot detect substituted paths as runtime dependencies in compressed ppd files. To ensure substituted paths are propagated as runtime dependencies, the script adds each substituted path to the variable `propagatedBuildInputs`. Since this might not be enough for multi-output packages, those paths are also written directly to `nix-support/propagated-build-inputs`. See the comment in `patch-ppd-hook.sh` for details. Finally, the setup hook comes with a small test that probes some edge cases with an artificial ppd file. References: * https://www.cups.org/doc/spec-ppd.html * general ppd file specification * lists some keys that may contain executable names or shell scripts * https://refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Printing/LSB-Printing/ppdext.html * lists some keys that may contain executable names or shell scripts * https://en.wikipedia.org/wiki/PostScript_Printer_Description#CUPS * lists the usual locations of ppd files
2021-07-07 15:38:55 +00:00
fixupOutputHooks+=(_patchPpdFileCommands4fixupOutputHooks)
# Install a hook for the `fixupPhase`:
# If the variable `ppdFileCommands` contains a list of
# executable names, the hook calls `patchPpdFileCommands`
# on each output's `/share/cups/model` and `/share/ppds`
# directories in order to replace calls to those executables.
_patchPpdFileCommands4fixupOutputHooks () {
[[ -n $ppdFileCommands ]] || return 0
if [[ -d $prefix/share/cups/model ]]; then
patchPpdFileCommands "$prefix/share/cups/model" $ppdFileCommands
fi
if [[ -d $prefix/share/ppds ]]; then
patchPpdFileCommands "$prefix/share/ppds" $ppdFileCommands
fi
}
# patchPpdFileCommands PPD-ROOT PROGNAME...
#
# Look for ppd files in the directory PPD-ROOT.
# Descend into subdirectories, even if they are symlinks.
# However, ignore ppd files that don't belong to the same
# prefix ($NIX_STORE/$package_name) as PPD-ROOT-DIR does,
# to avoid stepping into other package's directories.
# ppd files may be gzipped; if the are,
# uncompress them, later recompress them.
# Skip symlinks to ppd files.
# PPD-ROOT may also be a single ppd file.
#
# Look for the PROGNAME executable in outputs and `buildInputs`,
# then look for PROGNAME invocations in the ppd files,
# without path or with common paths like `/usr/bin/$PROGNAME`.
# Replace those invocations with an absolute path to the
# corresponding executable from the outputs or `buildInputs`.
# Executables are searched where CUPS would search them,
# i.e., in `/bin` and `/lib/cups/filter`.
#
# As soon as an executable's path is replaced as
# described above, the package containing the binary
# is added to the list of propagated build inputs.
# This ensures the executable's package is still
# recognized as runtime dependency of the ppd file
# even if the ppd file is compressed lateron.
#
# PROGNAME may not contain spaces or tabs.
# The function will also likely fail or produce
# broken results if PROGNAME contains characters that
# require shell or regex escaping (e.g. a backslash).
patchPpdFileCommands () {
local bin binnew binold binoldgrep cupspath path ppdroot ppdrootprefix
# we will store some temporary data here
pushd "$(mktemp -d --tmpdir patch-ppd-file-commands.XXXX)"
# remember the ppd root path
[[ "$1" == $NIX_STORE/* ]] # ensure it's a store directory
ppdroot=$1
shift # now "$@" is the list of binaries
ppdrootprefix=${ppdroot%"/${ppdroot#"$NIX_STORE"/*/}"}
# create `cupspath` (where we should look for binaries),
# with these priorities
# * outputs of current build before buildInputs
# * `/lib/cups/filter' before `/bin`
# * add HOST_PATH at end, so we don't miss anything
for path in $(getAllOutputNames); do
patchPpdFilesHook: new setup hook for absolute executable paths PostScript Printer Description (ppd) files describe printer features and capabilities. They are usually evaluated by CUPS to convert print jobs into a format suitable for a printer. The conversion is often accomplished by commands or even short shell scripts inside the ppd files. ppd files are included in many printer driver packages. Their scripts sometimes refer to other executables; some of them are more common (like `perl`), others are more exotic (like `rastertohp`). If an executable is called with its name alone, the effects of the ppd file depend on whether the executable is in the PATH of CUPS, and on the executable's version. If an executable is called with an absolut path (like `/usr/bin/perl`), it won't work at all in NixOS. The commit at hand adds a setup hook that uses the `fixupPhase` to substitute certain executable's invocations in pdd files with absolute paths. To use it, add the hook to `nativeBuildInputs` and provide a list of executable names in `ppdFileCommands`. Each executable must be available in the package that is being built, or in `buildInputs`. The setup hook's script then looks for ppd files in `share/cups/model` and `share/ppds` in each output, and replaces executable names with their absolute paths. If ppd files need to be patched in unorthodox locations or the setup hook needs to be invoked manually for other reasons, one may leave the list `ppdFileCommands` empty to avoid automatic processing of ppd files, then call the shell function `patchPpdFileCommands` directly. Details are described in the file `patch-ppd-hook.sh`. Notes on the motivation for this setup hook: Most packages in nixpkgs that provide ppd files do not patch those ppd files at all. This is not fatal when the executables are just called with their names since the user can add packages with the executables to `services.printing.drivers`. E.g. if the user adds `pkgs.perl`, then all ppd files that invoke `perl` will work as expected. Nevertheless, to make these ppd files independent of their execution environment, command invocations should be substituted with absolut paths into the nix store. This is similar to patching shebang lines so scripts can be called independently of having the interpreter in the PATH. The hook script in this commit is meant to support new packages `foomatic-db*` which will generate several thousands of ppd files referencing a plethora of different executables. During development of these packages, I realized that it's quite hard to patch ppd files in a robust way. While binary names like `rastertokpsl` seem to be sufficiently unique to be patched with `sed`, names like `date` or `gs` are hard to patch without producing "false positives", i.e., coincidental occurences of the executable's name that do *not* refer to the executable and should not be patched at all. As this problem also affects other packages, it seems reasonable to put a robust implementation in its own setup hook so that other packages can use it without much effort. Notes on the implementation: The ppd file format is far from trivial. The basic structure are key-value pairs; keys may occur multiple times. Only a small subset of keys may contain executable names or shell scripts in their values. Some values may span multiple lines; a linebreak might even occur in the middle of a token. Some executable names also occur in other keys by accident where they must not be substituted (e.g. `gs` or `date`). It is necessary to provide the list of command names that will be patched for two reasons: ppd files often contain "tokens" that might look like commands (e.g. "file" or "host") but aren't; these would erroneously get patched. Also, looking for everything that might be a command would slow down the patching process considerably. The implementation uses `awk` to detect keys that might contain executable names; only their values are treated for substitution. This avoids most cases of "overzealous" substitutions. Since values may span multiple lines, `sed` alone (while faster than `awk`) cannot focus its substitution capabilities on relevant keys. An elaborate set of regular expressions further helps to minimize the probability of "false positives". Several tricks are employed to speed up `awk`. Notably, relevant files are identified with `grep` before `awk` is applied to those files only. Note that the script probably cannot handle fancy command names (like spaces or backslashes as part of the name). Also, there are still edge cases that the script would mistakenly skip, e.g. if a shell script contains a line break in the middle of an executable's name; although ppd files permit such constellations, I have yet to see one. ppd files may be gzipped. The setup hook accepts gzipped ppd files: It decompresses them, substitutes paths, then recompresses them. However, Nix cannot detect substituted paths as runtime dependencies in compressed ppd files. To ensure substituted paths are propagated as runtime dependencies, the script adds each substituted path to the variable `propagatedBuildInputs`. Since this might not be enough for multi-output packages, those paths are also written directly to `nix-support/propagated-build-inputs`. See the comment in `patch-ppd-hook.sh` for details. Finally, the setup hook comes with a small test that probes some edge cases with an artificial ppd file. References: * https://www.cups.org/doc/spec-ppd.html * general ppd file specification * lists some keys that may contain executable names or shell scripts * https://refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Printing/LSB-Printing/ppdext.html * lists some keys that may contain executable names or shell scripts * https://en.wikipedia.org/wiki/PostScript_Printer_Description#CUPS * lists the usual locations of ppd files
2021-07-07 15:38:55 +00:00
addToSearchPath cupspath "${!path}/lib/cups/filter"
addToSearchPath cupspath "${!path}/bin"
done
for path in ${pkgsHostTarget+"${pkgsHostTarget[@]}"}; do
addToSearchPath cupspath "$path/lib/cups/filter"
addToSearchPath cupspath "$path/bin"
done
while read -r -d : path; do
addToSearchPath cupspath "$path"
done <<< "${HOST_PATH:+"${HOST_PATH}:"}"
# create list of compressed ppd files
# so we can recompress them later
find -L "$ppdroot" -type f -iname '*.ppd.gz' '!' -xtype l -print0 > gzipped
# decompress gzipped ppd files
echo "patchPpdFileCommands: decompressing $(grep -cz '^' < gzipped) gzipped ppd file(s) in $ppdroot"
xargs -0r -n 64 -P "$NIX_BUILD_CORES" gunzip < gzipped
# create list of all ppd files to be checked
find -L "$ppdroot" -type f -iname '*.ppd' '!' -xtype l -print0 > ppds
for bin in "$@"; do
# discover new path
binnew=$(PATH=$cupspath '@which@/bin/which' "$bin")
echo "patchPpdFileCommands: located binary $binnew"
# for each binary, we look for the name itself, but
# also for a couple of common paths that might be used
for binold in {/usr,}/{lib/cups/filter,sbin,bin}/"$bin" "$bin"; do
# escape regex characters in the old command string
binoldgrep=$(sed 's,[]$.*[\^],\\&,g' <<< "$binold")
# ...and surround old command with some regex
# that singles out shell command invocations
# to avoid replacing other strings that might contain the
# command name by accident (like "perl" in "perl-script")
binoldgrep='\(^\|[;&| '$'\t''"`(]\)'"$binoldgrep"'\($\|[);&| '$'\t''"`<>]\)'
# this string is used to *quickly* filter out
# unaffected files before the (slower) awk script runs;
# note that a similar regex is build in the awk script;
# if `binoldgrep` is changed, the awk script should also be checked
# create list of likely affected files
# (might yield exit status != 0 if there are no matches)
xargs -0r grep -lZ "$binoldgrep" < ppds > ppds-to-patch || true
echo "patchPpdFileCommands: $(grep -cz '^' < ppds-to-patch) ppd file(s) contain $binold"
# actually patch affected ppd files with awk;
# this takes some time but can be parallelized;
# speed up with LC_ALL=C, https://stackoverflow.com/a/33850386
LC_ALL=C xargs -0r -n 64 -P "$NIX_BUILD_CORES" \
awk -i inplace -v old="${binold//\\/\\\\}" -v new="${binnew//\\/\\\\}" -f "@awkscript@" \
< ppds-to-patch
done
# create list of affected files
xargs -0r grep -lZF "$binnew" < ppds > patched-ppds || true
echo "patchPpdFileCommands: $(grep -cz '^' < patched-ppds) ppd file(s) patched with $binnew"
# if the new command is contained in a file,
# remember the new path so we can add it to
# the list of propagated dependencies later
if [[ -s patched-ppds ]]; then
printf '%s\0' "${binnew%"/${binnew#"${NIX_STORE}"/*/}"}" >> dependencies
fi
done
# recompress ppd files that have been decompressed before
echo "patchPpdFileCommands: recompressing $(grep -cz '^' < gzipped) gzipped ppd file(s)"
# we can't just hand over the paths of the uncompressed files
# to gzip as it would add the lower-cased extension ".gz"
# even for files where the original was named ".GZ"
xargs -0r -n 1 -P "$NIX_BUILD_CORES" \
"$SHELL" -c 'gzip -9nS ".${0##*.}" "${0%.*}"' \
< gzipped
# enlist dependencies for propagation;
# this is needed in case ppd files are compressed later
# (Nix won't find dependency paths in compressed files)
if [[ -s dependencies ]]; then
# weed out duplicates from the dependency list first
sort -zu dependencies > sorted-dependencies
mkdir -p "$ppdrootprefix/nix-support"
while IFS= read -r -d '' path; do
printWords "$path" >> "$ppdrootprefix/nix-support/propagated-build-inputs"
# stdenv writes it's own `propagated-build-inputs`,
# based on the variable `propagatedBuildInputs`,
# but only to one output (`outputDev`).
# So we also add our dependencies to that variable.
# If our file survives as written above, great!
# If stdenv overwrits it,
# our dependencies will still be added to the file.
# The end result might contain too many
# propagated dependencies for multi-output packages,
# but never a broken package.
propagatedBuildInputs+=("$path")
done < sorted-dependencies
fi
popd
}