nix/mk/run-test.sh

#!/usr/bin/env bash

set -eu -o pipefail

red=""
green=""
yellow=""
normal=""

test=$1

dir="$(dirname "${BASH_SOURCE[0]}")"
source "$dir/common-test.sh"

post_run_msg="ran test $test..."
if [ -t 1 ]; then
    red="[31;1m"
    green="[32;1m"
    yellow="[33;1m"
    normal="[m"
fi

run_test () {
    (init_test 2>/dev/null > /dev/null)
    log="$(run_test_proper 2>&1)" && status=0 || status=$?
}

run_test

if [ $status -eq 0 ]; then
  echo "$post_run_msg [${green}PASS$normal]"
elif [ $status -eq 99 ]; then
  echo "$post_run_msg [${yellow}SKIP$normal]"
else
  echo "$post_run_msg [${red}FAIL$normal]"
  echo "$log" | sed 's/^/    /'
  exit "$status"
fi
-												Make `./mk/run-test.sh` work by itself; add `mk/debug-test.sh`

First, logic is consolidated in the shell script instead of being spread
between them and makefiles. That makes understanding what is going on a
little easier.

This would not be super interesting by itself, but it gives us a way to
debug tests more easily. *That* in turn I hope is much more compelling.
See the updated manual for details.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

											
										
										
											2022-12-16 01:17:08 +00:00
+								#!/usr/bin/env bash
-												Run the tests in parallel

Cause the time needed to run the testsuite to drop from ~4mins to ~40s

											
										
										
											2020-07-02 09:34:15 +00:00
-												Harden tests' bash

Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.

 - `set -u` catches the use of undefined variables
 - `set -o pipefail` catches failures (like `set -e`) earlier in the
   pipeline.

This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.

There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.

To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.

`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.

`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.

`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

											
										
										
											2021-12-09 15:26:46 +00:00
+								set -eu -o pipefail
-												Run the tests in parallel

Cause the time needed to run the testsuite to drop from ~4mins to ~40s

											
										
										
											2020-07-02 09:34:15 +00:00
 								red=""
 								green=""
 								yellow=""
 								normal=""
-												Make `./mk/run-test.sh` work by itself; add `mk/debug-test.sh`

First, logic is consolidated in the shell script instead of being spread
between them and makefiles. That makes understanding what is going on a
little easier.

This would not be super interesting by itself, but it gives us a way to
debug tests more easily. *That* in turn I hope is much more compelling.
See the updated manual for details.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

											
										
										
											2022-12-16 01:17:08 +00:00
+								test=$1
 								dir="$(dirname "${BASH_SOURCE[0]}")"
 								source "$dir/common-test.sh"
 								post_run_msg="ran test $test..."
-												Run the tests in parallel

Cause the time needed to run the testsuite to drop from ~4mins to ~40s

											
										
										
											2020-07-02 09:34:15 +00:00
+								if [ -t 1 ]; then
 								    red="[31;1m"
 								    green="[32;1m"
 								    yellow="[33;1m"
 								    normal="[m"
 								fi
-												Acknowledge that the macOS tests are flaky

Restart the tests (at most once) on `unexpected EOF` errors.

This is truly ugly, but might prevent half of the CI runs to fail
because of https://github.com/NixOS/nix/issues/3605

											
										
										
											2022-03-01 13:59:12 +00:00
 								run_test () {
-												Make `./mk/run-test.sh` work by itself; add `mk/debug-test.sh`

First, logic is consolidated in the shell script instead of being spread
between them and makefiles. That makes understanding what is going on a
little easier.

This would not be super interesting by itself, but it gives us a way to
debug tests more easily. *That* in turn I hope is much more compelling.
See the updated manual for details.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

											
										
										
											2022-12-16 01:17:08 +00:00
+								    (init_test 2>/dev/null > /dev/null)
-												Harden tests' bash

Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.

 - `set -u` catches the use of undefined variables
 - `set -o pipefail` catches failures (like `set -e`) earlier in the
   pipeline.

This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.

There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.

To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.

`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.

`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.

`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

											
										
										
											2021-12-09 15:26:46 +00:00
+								    log="$(run_test_proper 2>&1)" && status=0 || status=$?
-												Acknowledge that the macOS tests are flaky

Restart the tests (at most once) on `unexpected EOF` errors.

This is truly ugly, but might prevent half of the CI runs to fail
because of https://github.com/NixOS/nix/issues/3605

											
										
										
											2022-03-01 13:59:12 +00:00
+								}
-												Make `./mk/run-test.sh` work by itself; add `mk/debug-test.sh`

First, logic is consolidated in the shell script instead of being spread
between them and makefiles. That makes understanding what is going on a
little easier.

This would not be super interesting by itself, but it gives us a way to
debug tests more easily. *That* in turn I hope is much more compelling.
See the updated manual for details.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

											
										
										
											2022-12-16 01:17:08 +00:00
+								run_test
-												Acknowledge that the macOS tests are flaky

Restart the tests (at most once) on `unexpected EOF` errors.

This is truly ugly, but might prevent half of the CI runs to fail
because of https://github.com/NixOS/nix/issues/3605

											
										
										
											2022-03-01 13:59:12 +00:00
-												Run the tests in parallel

Cause the time needed to run the testsuite to drop from ~4mins to ~40s

											
										
										
											2020-07-02 09:34:15 +00:00
+								if [ $status -eq 0 ]; then
 								  echo "$post_run_msg [${green}PASS$normal]"
 								elif [ $status -eq 99 ]; then
 								  echo "$post_run_msg [${yellow}SKIP$normal]"
 								else
 								  echo "$post_run_msg [${red}FAIL$normal]"
 								  echo "$log" | sed 's/^/    /'
 								  exit "$status"
 								fi