When IPXE tests were added, an option was added for configuring only
the frontend, and the backend configuration was dropped entirely. This
caused most installer tests to fail.
See #49441 for an earlier attempt, which was subsequently reverted. I am
assuming that doubling the time will be sufficient if the machine is
overloaded since so many of the tests already pass at 5 minutes, while
still not holding back failures for needlessly long.
The ability to specify "-drive if=scsi" has been removed in QEMU version
2.12 (introduced in 3e3b39f173).
Quote from https://wiki.qemu.org/ChangeLog/2.12#Incompatible_changes:
> The deprecated way of configuring SCSI devices with "-drive if=scsi"
> on x86 has been removed. Use an appropriate SCSI controller together
> "-device scsi-hd" or "-device scsi-cd" and a corresponding "-blockdev"
> parameter instead.
So whenever the diskInterface is "scsi" we use the new way to specify
the drive and fall back to the deprecated way for the time being. The
reason why I'm not using the new way for "virtio" and "ide" as well is
because there is no simple generic way anymore to specify these.
This also turns the type of the virtualisation.qemu.diskInterface option
to be an enum, so the user knows which values are allowed but we can
also make sure the right value is provided to prevent typos.
I've tested this against a few non-disk-related NixOS VM tests but also
the installer.grub1 test (because it uses "ide" as its drive interface),
the installer.simple test (just to be sure it still works with
"virtio") and all the tests in nixos/tests/boot.nix.
In order to be able to run the grub1 test I had to go back to
8b1cf100cd (which is a known commit where
that test still works) and apply the QEMU update and this very commit,
because right now the test is broken.
Apart from the tests here in nixpkgs, I also ran another[1] test in
another repository which uses the "scsi" disk interface as well (in
comparison to most of the installer tests, this one actually failed
prior to this commit).
All of them now succeed.
[1]: 9b5a119972/tests/system/kernel/bfq.nix
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edostra, @grahamc, @dezgeg, @abbradar, @ts468
It is quite complicated to test services using the test-driver when
declaring user services with `systemd.user.services` such as many
X11-based services like `xautolock.service`.
This change adds an optional `$user` parameter to each systemd-related
function in the test-driver and runs `systemctl --user` commands using
`su -l $user -c ...` and sets the `XDG_RUNTIME_DIR` variable
accordingly and a new function named `systemctl` which is able to run a
systemd command with or without a specified user.
The change can be confirmed with a simple VM declaration like this:
```
import ./nixos/tests/make-test.nix ({ pkgs, lib }:
with lib;
{
name = "systemd-user-test";
nodes.machine = {
imports = [ ./nixos/tests/common/user-account.nix ];
services.xserver.enable = true;
services.xserver.displayManager.auto.enable = true;
services.xserver.displayManager.auto.user = "bob";
services.xserver.xautolock.enable = true;
};
testScript = ''
$machine->start;
$machine->waitForX;
$machine->waitForUnit("xautolock.service", "bob");
$machine->stopJob("xautolock.service", "bob");
$machine->startJob("xautolock.service", "bob");
$machine->systemctl("list-jobs --no-pager", "bob");
$machine->systemctl("show 'xautolock.service' --no-pager", "bob");
'';
})
```
machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d'
machine: exit status 0
machine: Last chance to match /(?^:dfiirst configuration)/ on the the window list, which currently contains:
machine: [i3 con] container around 0xf8a5f0, i3: first configuration, [i3 con] floatingcon around 0xf8c260, [i3 con] container around 0xf8a380, i3bar for output Virtual-1, [i3 con] bottom dockarea Virtual-1, [i3 con] workspace 1, [i3 con] content Virtual-1, [i3 con] top dockarea Virtual-1, [i3 con] output Virtual-1, [i3 con] workspace __i3_scratch, [i3 con] content __i3, [i3 con] pseudo-output __i3, i3
machine: Last chance to match /(?^:BALICE)/ on the screen, which currently contains:
machine: performing optical character recognition
machine: sending monitor command: screendump /tmp/nix-build-vm-test-run-sddm.drv-0/ocrin.ppm
machine: Session Layout
O O
0 1 : 0 9
Wednesday, June 21, 2017
|_ I
Select your user and enter password
If the test has not passed yet, on the last attempt it now outputs:
machine: Last chance to match /logine: / on TTY2, which currently contains:
machine: running command: fold -w$(stty -F /dev/tty2 size | awk '{print $2}') /dev/vcs2
machine: exit status 0
machine:
<<< Welcome to NixOS 17.09.git.a804ef4 (x86_64) - tty2 >>>
machine login:
to help debug the problem. Notice the "logine" typo in my check.
First of all, we're now using ImageMagick to improve the screenshot so
that Tesseract has an esier time to recognize the text. The resulting
image of this post-processing is a scaled up black-and-white version
with the backgrounds almost entirely removed and the text edges a bit
blurred, so the screen shots now more or less resemble an image from a
scanner rather. This is what Tesseract is trained for by default.
As mentioned in the previous commit we now also use Tesseract 4, which
further improves the quality of text recognition.
I've spent countless hours just to test different postprocessing
variants and testing what works best for our tests and this is the one
that worked best so far. It's certainly not perfect and I'd like to
avoid the scaling step but we're way better off than before.
In addition to this, the OCR process is now done without an intermediate
file, solely using pipes.
I've tested this using the following VM tests which have OCR enabled:
* nixos/tests/chromium.nix -A stable
* nixos/tests/emacs-daemon.nix
* nixos/tests/installer.nix -A luksroot
* nixos/tests/lightdm.nix
* nixos/tests/plasma5.nix
* nixos/tests/sddm.nix
All of the tests still succeed and comparing some of the recognition
results to the earlier results it now also detects a lot more text than
before this commit.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
A long-time issue and one of the reasons I've never used that function
before. So let's remove that todo-comment and escape the contents
properly.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
Regression introduced by d84741a4bf.
The mentioned commit actually is a good thing, because we now get the
output from the X session.
Unfortunately, for the i3wm test, the i3-config-wizard prints out the
raw keyboard symbols directly coming from xcb, so the output isn't
necessarily proper UTF-8.
As the XML::Writer already expects valid UTF-8 input, we assume that
everything that comes into sanitise() will be UTF-8 from the start. So
we just decode() it using FB_DEFAULT as the check argument so that
every invalid character is replaced by the unicode replacement
character:
https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character
We simply re-oncode it again afterwards and return it, so we should
always get out valid UTF-8 in the log XML.
For more information about FB_DEFAULT and FB_CROAK, have a look at:
http://search.cpan.org/~dankogai/Encode-2.84/Encode.pm#Handling_Malformed_Data
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
By default this is now enabled, and it has to be explicitely enabled
using "enableOCR = true". If it is set to false, any usage of
getScreenText or waitForText will fail with an error suggesting to pass
enableOCR.
This should get rid of the rather large dependency on tesseract which
we don't need for most tests.
Note, that I'm using system("type -P") here to check whether tesseract
is in PATH. I know it's a bashism but we already have other bashisms
within the test scripts and we also run it with bash, so IMHO it's not a
problem here.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
As promised in the previous commit, this can be used similarly to
$machine->waitForWindow, where you supply a regular expression and it's
retrying OCR until the regexp matches.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Basically, this creates a screenshot and throws tesseract at it to
recognize the characters from the screenshot. In order to produce a
result that is well enough, we're using lanczos scaling and scale the
image up to 400% of its original size.
This provides the base functionality for a new Machine method which will
be called waitForText. I originally had that idea long ago when writing
the VM tests for VirtualBox and Chromium, but thought it would be
disproportionate to the case.
The downside however is that VM tests now depend on tesseract, but given
the average runtime of our tests it really shouldn't have a too big
impact and it's only a runtime dependency after all.
Another issue is that the OCR process takes quite some time to finish,
but IMHO it's better (as in more deterministic) than to rely on sleep().
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The current way test reports get jquery,
src="//ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"
only works when getting reports over http:// or https://, not file://.
Change it so that it works for all protocols by using a local copy of
jquery.
This fixes the issue where locally created and browsed test reports
cannot be navigated properly; clicking the '+' symbol to expand
sub-sections doesn't work.
This reverts commit 4e6eae45ee. It
breaks running the test driver interactively (in that it causes all
VMs to be started immediately, which is not always what you wnat).
