nordic-dev.net/nixpkgs
nordic-dev.net/nixpkgs
2
0
Fork 0
mirror of https://github.com/NixOS/nixpkgs.git synced 2024-12-19 04:03:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
251,057 Commits 186 Branches 125 Tags 7.5 GiB
523debafc4
Commit Graph

19059 Commits

Author SHA1 Message Date
Graham Christensen
74a577b293
create-amis: improve wording around the service name's IAM role 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2020-10-30 12:40:17 -04:00
Graham Christensen
ece5c0f304
stage-1: modprobe ext{2,3,4} before resizing 
I noticed booting a system with an ext4 root which expanded to 5T took
quite a long time (12 minutes in some cases, 43(!) in others.)

I changed stage-1 to run `resize2fs -d 62` for extra debug output and
timing information. It revealed the adjust_superblock step taking
almost all of the time:

    [Fri Oct 30 11:10:15 UTC 2020] zero_high_bits_in_metadata: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
    [Fri Oct 30 11:21:09 UTC 2020] adjust_superblock: Memory used: 396k/4556k (295k/102k), time: 654.21/ 0.59/ 5.13

but when I ran resize2fs on a disk with the identical content growing
to the identical target size, it would only take about 30 seconds. I
looked at what happened between those two steps in the fast case with
strace and found:

```
   235	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1795}, ru_stime={tv_sec=0, tv_usec=3590}, ...}) = 0
   236	write(1, "zero_high_bits_in_metadata: Memo"..., 84zero_high_bits_in_metadata: Memory used: 132k/0k (72k/61k), time:  0.00/ 0.00/ 0.00
   237	) = 84
   238	gettimeofday({tv_sec=1604061278, tv_usec=480147}, NULL) = 0
   239	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1802}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   240	gettimeofday({tv_sec=1604061278, tv_usec=480192}, NULL) = 0
   241	mmap(NULL, 2564096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa3c7355000
   242	access("/sys/fs/ext4/features/lazy_itable_init", F_OK) = 0
   243	brk(0xf85000)                           = 0xf85000
   244	brk(0xfa6000)                           = 0xfa6000
   245	gettimeofday({tv_sec=1604061278, tv_usec=538828}, NULL) = 0
   246	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58720}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   247	write(1, "adjust_superblock: Memory used: "..., 79adjust_superblock: Memory used: 396k/2504k (305k/92k), time:  0.06/ 0.06/ 0.00
   248	) = 79
   249	gettimeofday({tv_sec=1604061278, tv_usec=539119}, NULL) = 0
   250	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58812}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   251	gettimeofday({tv_sec=1604061279, tv_usec=939}, NULL) = 0
   252	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=520411}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   253	write(1, "fix_uninit_block_bitmaps 2: Memo"..., 88fix_uninit_block_bitmaps 2: Memory used: 396k/2504k (305k/92k), time:  0.46/ 0.46/ 0.00
   254	) = 88
```

In particular the access to /sys/fs seemed interesting. Looking
at the source of resize2fs:

```
[root@ip-172-31-22-182:~/e2fsprogs-1.45.5]# rg -B2 -A1 /sys/fs/ext4/features/lazy_itable_init .
./resize/resize2fs.c
923-	if (getenv("RESIZE2FS_FORCE_LAZY_ITABLE_INIT") ||
924-	    (!getenv("RESIZE2FS_FORCE_ITABLE_INIT") &&
925:	     access("/sys/fs/ext4/features/lazy_itable_init", F_OK) == 0))
926-		lazy_itable_init = 1;
```

I confirmed /sys is mounted, and then found a bug suggesting the
ext4 module is maybe not loaded:
https://bugzilla.redhat.com/show_bug.cgi?id=1071909

My home server doesn't have ext4 loaded and had 3T to play with, so
I tried (and succeeded with) replicating the issue locally:

```
[root@kif:/scratch]# lsmod | grep -i ext

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 560a4a8f-93dc-40cc-97a5-f10049bf801f
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m2.261s
user	0m0.000s
sys	0m0.025s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3802.28MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
```

here it got stuck for quite some time ... straceing this 20 minutes in revealed this in a tight loop:

```
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
fallocate(3, FALLOC_FL_ZERO_RANGE, 2222649901056, 2097152) = 0
fsync(3)                                = 0
```

it finally ended 43(!) minutes later:

```
adjust_superblock: Memory used: 264k/3592k (210k/55k), time: 2554.03/ 0.16/15.07
fix_uninit_block_bitmaps 2: Memory used: 264k/3592k (210k/55k), time:  0.16/ 0.16/ 0.00
blocks_to_move: Memory used: 264k/3592k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.05/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 18.68MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.35/16.35/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.04/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 22.80MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 2570.90/16.68/15.07
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.00MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	43m1.943s
user	0m16.761s
sys	0m15.069s
```

I then cleaned up and recreated the zvol, loaded the ext4 module, created the ext4 fs,
resized the volume, and resize2fs'd and it went quite quickly:

```
[root@kif:/scratch]# zfs destroy rpool/scratch/ext4

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# modprobe ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 5b415f2f-a8c4-4ba0-ac1d-78860de77610
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m1.013s
user	0m0.001s
sys	0m0.023s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3389.83MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
adjust_superblock: Memory used: 264k/1540k (210k/55k), time:  0.02/ 0.02/ 0.00
fix_uninit_block_bitmaps 2: Memory used: 264k/1540k (210k/55k), time:  0.15/ 0.15/ 0.00
blocks_to_move: Memory used: 264k/1540k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.01/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 157.11MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00

calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.20/16.20/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 5319.15MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 16.45/16.38/ 0.00
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.06MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	0m17.908s
user	0m16.386s
sys	0m0.079s
```

Success!
 2020-10-30 12:18:23 -04:00
Graham Christensen
a179781696
stage-1: add datestamps to logs 
When the stage-1 logs get imported in to the journal, they all get
loaded with the same timestamp. This makes it difficult to identify
what might be taking a long time in early boot.
 2020-10-30 12:16:35 -04:00
Graham Christensen
2bf1fc0345
create-amis: allow customizing the service role name 
The complete setup on the AWS end can be configured
with the following Terraform configuration. It generates
a ./credentials.sh which I just copy/pasted in to the
create-amis.sh script near the top. Note: the entire stack
of users and bucket can be destroyed at the end of the
import.

    variable "region" {
      type = string
    }
    variable "availability_zone" {
      type = string
    }

    provider "aws" {
      region = var.region
    }

    resource "aws_s3_bucket" "nixos-amis" {
      bucket_prefix = "nixos-amis-"
      lifecycle_rule {
        enabled = true
        abort_incomplete_multipart_upload_days = 1
        expiration {
          days = 7
        }
      }
    }

    resource "local_file" "credential-file" {
      file_permission = "0700"
      filename = "${path.module}/credentials.sh"
      sensitive_content = <<SCRIPT
    export service_role_name="${aws_iam_role.vmimport.name}"
    export bucket="${aws_s3_bucket.nixos-amis.bucket}"
    export AWS_ACCESS_KEY_ID="${aws_iam_access_key.uploader.id}"
    export AWS_SECRET_ACCESS_KEY="${aws_iam_access_key.uploader.secret}"
    SCRIPT
    }

    # The following resources are for the *uploader*
    resource "aws_iam_user" "uploader" {
      name = "nixos-amis-uploader"
    }

    resource "aws_iam_access_key" "uploader" {
      user = aws_iam_user.uploader.name
    }

    resource "aws_iam_user_policy" "upload-to-nixos-amis" {
      user = aws_iam_user.uploader.name

      policy = data.aws_iam_policy_document.upload-policy-document.json
    }

    data "aws_iam_policy_document" "upload-policy-document" {
      statement {
        effect = "Allow"

        actions = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
        ]

        resources = [
          aws_s3_bucket.nixos-amis.arn
        ]
      }

      statement {
        effect = "Allow"

        actions = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
        ]

        resources = [
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }

      statement {
        effect = "Allow"
        actions = [
          "ec2:ImportSnapshot",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:RegisterImage",
          "ec2:DescribeImages"
        ]
        resources = [
          "*"
        ]
      }
    }

    # The following resources are for the *vmimport service user*
    # See: https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role
    resource "aws_iam_role" "vmimport" {
      assume_role_policy = data.aws_iam_policy_document.vmimport-trust.json
    }

    resource "aws_iam_role_policy" "vmimport-access" {
      role = aws_iam_role.vmimport.id
      policy = data.aws_iam_policy_document.vmimport-access.json
    }

    data "aws_iam_policy_document" "vmimport-access" {
      statement {
        effect = "Allow"
        actions = [
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
        ]
        resources = [
          aws_s3_bucket.nixos-amis.arn,
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }
      statement {
        effect = "Allow"
        actions = [
          "ec2:ModifySnapshotAttribute",
          "ec2:CopySnapshot",
          "ec2:RegisterImage",
          "ec2:Describe*"
        ]
        resources = [
          "*"
        ]
      }
    }

    data "aws_iam_policy_document" "vmimport-trust" {
      statement {
        effect = "Allow"
        principals {
          type = "Service"
          identifiers = [ "vmie.amazonaws.com" ]
        }

        actions = [
          "sts:AssumeRole"
        ]

        condition {
          test = "StringEquals"
          variable = "sts:ExternalId"
          values = [ "vmimport" ]
        }
      }
    }
 2020-10-30 12:12:08 -04:00
Graham Christensen
e253de8a77
create-amis.sh: log the full response if describing the import snapshot tasks fails 2020-10-30 12:08:01 -04:00
Graham Christensen
f92a883ddb
nixos ec2/create-amis.sh: shellcheck: $ is not needed in arithmetic 2020-10-30 12:08:01 -04:00
Graham Christensen
7dac8470cf
nixos ec2/create-amis.sh: shellcheck: explicitly make the additions to block_device_mappings single strings 2020-10-30 12:08:00 -04:00
Graham Christensen
a66a22ca54
nixos ec2/create-amis.sh: shellcheck: read without -r mangles backslashes 2020-10-30 12:08:00 -04:00
Graham Christensen
baf7ed3f24
nixos ec2/create-amis.sh: shellcheck: SC2155: Declare and assign separately to avoid masking return values. 2020-10-30 12:07:59 -04:00
Graham Christensen
f5994c208d
nixos ec2/create-amis.sh: shellcheck: quote state_dir reference 2020-10-30 12:07:59 -04:00
Graham Christensen
c76692192a
nixos ec2/create-amis.sh: shellcheck: quote region references 2020-10-30 12:07:49 -04:00
Timo Kaufmann
83f48e8348
Merge pull request #95011 from Atemu/undervolt-pl 
undervolt: expose power limits as Nixopts
 2020-10-30 09:32:50 +01:00
Michele Guerini Rocco
1102a46ffe
Merge pull request #101724 from pickfire/patch-3 
fontdir: add ttc to font regex
 2020-10-30 08:41:34 +01:00
Benjamin Hipple
e00752079e
Merge pull request #102018 from 1000101/blockbook-frontend 
blockbook-frontend: fix&update extraConfig example
 2020-10-29 22:30:07 -04:00
Florian Klink
b8d59e93c8 nixos/networkd: allow RouteMetric= in [DHCPv6] section 2020-10-29 19:47:42 +01:00
talyz
89e83833af
nixos/keycloak: Add support for MySQL and external DBs with SSL 
- Add support for using MySQL as an option to PostgreSQL.
- Enable connecting to external DBs with SSL
- Add a database port config option
 2020-10-29 12:47:10 +01:00
talyz
d1d3c86c70
rl-2103: Note the addition of the Keycloak service 2020-10-29 12:08:06 +01:00
talyz
c6e4388449
nixos/keycloak: Add documentation 2020-10-29 12:08:01 +01:00
talyz
fe5a16aee6
nixos/keycloak: Document internal functions 2020-10-29 12:07:55 +01:00
talyz
31fe90d6ef
nixos/keycloak: Add test 2020-10-29 12:07:49 +01:00
1000101
4b8611c959 blockbook-frontend: fix&update extraConfig example 2020-10-29 11:41:41 +01:00
Philipp
fc856b89e5
nixos/murmur: add murmur group, don't run as nogroup 
fixes #101980
 2020-10-29 10:32:04 +01:00
Martin Weinelt
55746e0a4b
Merge pull request #98187 from mweinelt/nixos/babeld 
nixos/babeld: lock down service
 2020-10-29 01:24:11 +01:00
Minijackson
3fce272478 nixos/shiori: harden service with systemd 2020-10-28 20:46:30 +01:00
Maximilian Bosch
ca45bb574d
nixos/rl-2009: minor typo fix 2020-10-28 19:38:28 +01:00
Thomas Depierre
63caecee7d riak-cs: delete 2020-10-28 19:31:33 +01:00
Linus Heckemann
2b06415ca1
Merge pull request #101370 from m1cr0man/ssl-test-certs 
nixos/acme: Permissions and tests fixes
 2020-10-28 17:21:57 +01:00
Andreas Rammhold
c127653b72
Merge pull request #101887 from jonringer/minor-release-notes-adjustment 
nixos/docs/rl-2009.xml: grafana: description, example agreement
 2020-10-28 02:38:55 +01:00
Jonathan Ringer
3963954fc8
nixos/docs/rl-2009.xml: grafana: description, example agreement 2020-10-27 17:50:39 -07:00
Markus S. Wamser
a0cc1243cc doc: 20.09 release notes: remove duplicate service list entry 
opt-services.foldingathome.enable was listed twice
 2020-10-27 13:43:44 -07:00
davidak
4166a767de doc: improve 20.09 release notes 2020-10-27 21:11:22 +01:00
talyz
513599a6d7
nixos/keycloak: Init 2020-10-27 19:01:26 +01:00
AmineChikhaoui
8cae6703ef
ec2-amis: add stable NixOS 20.09 AMIs 
Fixes #101694
 2020-10-27 08:52:15 -04:00
WORLDofPEACE
5a08ab936b rl-2009: release on a Tuesday 
Because hydra took it's good old time
 2020-10-27 03:03:43 -04:00
Ryan Mulligan
178d373a8a
Merge pull request #83687 from primeos/wshowkeys 
wshowkeys: init at 2020-03-29
 2020-10-26 18:55:16 -07:00
WORLDofPEACE
d1b239703c
Merge pull request #101811 from jonringer/rl-2009-contributions 
release-notes-2009: add contributions section
 2020-10-26 21:49:20 -04:00
Jonathan Ringer
51ca426eb5
release-notes-2009: add contributions section 2020-10-26 18:36:12 -07:00
Andreas Rammhold
1dc37370c4
Merge pull request #101805 from andir/unbreak-tarball-job 
nixos/tests: fix wrong inherit that passes on the nodes attrs
 2020-10-27 01:29:36 +01:00
Jonathan Ringer
366bebd53a README.md: update stable release links 2020-10-26 20:10:29 -04:00
Andreas Rammhold
eba8f5425f
nixos/tests: fix wrong inherit that passes on the nodes attrs 
The hydra tarball step would fail due to the nodes attribute not being
properly inherited. Since we can't execute all the tests and release
steps locally anymore (thanks to the JSONification and faster hydra
eval) these errors will probably keep in appearing.

This is hopefully the last of those introduced by me test runner
refactoring.

Error was seen on hydra (https://hydra.nixos.org/build/129282411):
> unpacking sources
> unpacking source archive /nix/store/bp95x52h6nv3j8apxrryyj2rviw682k1-source
> source root is source
> patching sources
> autoconfPhase
> No bootstrap, bootstrap.sh, configure.in or configure.ac. Assuming this is not an GNU Autotools package.
> configuring
> release name is nixpkgs-21.03pre249116.1088f059401
> git-revision is 1088f05940
> building
> no Makefile, doing nothing
> running tests
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> checking Nixpkgs on i686-linux
> checking Nixpkgs on x86_64-linux
> checking Nixpkgs on x86_64-darwin
> checking eval-release.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> checking find-tarballs.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> error: while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:6:1, called from undefined position:
> while evaluating 'operator' at /build/source/maintainers/scripts/find-tarballs.nix:27:16, called from undefined position:
> while evaluating 'immediateDependenciesOf' at /build/source/maintainers/scripts/find-tarballs.nix:39:29, called from /build/source/maintainers/scripts/find-tarballs.nix:27:44:
> while evaluating anonymous function at /build/source/lib/attrsets.nix:234:10, called from undefined position:
> while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:40:37, called from /build/source/lib/attrsets.nix:234:16:
> while evaluating 'derivationsIn' at /build/source/maintainers/scripts/find-tarballs.nix:42:19, called from /build/source/maintainers/scripts/find-tarballs.nix:40:40:
> while evaluating 'canEval' at /build/source/maintainers/scripts/find-tarballs.nix:48:13, called from /build/source/maintainers/scripts/find-tarballs.nix:43:9:
> while evaluating the attribute 'nodes' at /build/source/nixos/lib/testing-python.nix:195:23:
> attribute 'nodes' missing, at /build/source/nixos/lib/testing-python.nix:193:16
> build time elapsed:  0m0.122s 0m0.043s 17m51.526s 0m56.668s
> builder for '/nix/store/96rk3c74vrk6m3snm7n6jhis3j640pn4-nixpkgs-tarball-21.03pre249116.1088f059401.drv' failed with exit code 1
 2020-10-27 00:10:31 +01:00
Tim Steinbach
c851af868f
docker-edge: Fix test 2020-10-26 16:25:37 -04:00
WORLDofPEACE
ace69f768b Revert "nixos/pantheon: install nixos wallpaper" 
This reverts commit 5100e4f250.

Fixes https://github.com/NixOS/nixpkgs/issues/100293
Though it's only a workaround for now.
See https://github.com/elementary/switchboard-plug-pantheon-shell/issues/246#issuecomment-716713218
We trigger the broken scenario where we have two subdirectories. Reverting
that commit undoes this.
 2020-10-26 13:45:19 -04:00
Nick Hu
921287e7f0
Merge pull request #97726 from NickHu/pam_gnupg 
pam: add support for pam_gnupg
 2020-10-26 15:27:13 +00:00
Andreas Rammhold
1088f05940
Merge pull request #101598 from andir/nixos-build-vms-qemu 
nixos/tests: follow-up to the closure reduction PR
 2020-10-26 14:19:45 +01:00
rnhmjoj
9e04bba0af nixos/dnscrypt-wrapper: fix key rotation script 
Fix an error in the validation code when the public key is in a
nonstandard location. The check command fails and the key is
incorrectly assumed to be expiring.
 2020-10-26 13:07:49 +01:00
Jörg Thalheim
dfaa313d43
Merge pull request #101737 from aneeshusa/nginx-allow-unsetting-ssl_ciphers 
nixos/nginx: Allow unsetting ssl_ciphers
 2020-10-26 06:41:19 +01:00
Aneesh Agrawal
924035bb97 nixos/nginx: Allow unsetting ssl_ciphers 
When using the Modern config from the Mozilla SSL config generator,
the `ssl_ciphers` parameter does not need to be set
as only TLSv1.3 is permitted and all of its ciphers are reasonable.
 2020-10-26 00:35:29 -04:00
Jörg Thalheim
b7a2a5f967
nixos/nextcloud: fix several php endpoints 2020-10-26 05:06:43 +01:00
Klemens Nanni
0b8a6e787c
nixos/avahi: Enable IPv6 by default 
Treat it the same as IPv4 (I'm tempted to disable IPv4 by default);
this is the only option I still need to set manually to enjoy IPv6-only
networks including printer discovery!
 2020-10-26 04:06:26 +01:00
Ivan Tham
f6136d06ff
fontdir: add ttc to font regex 
.ttc fonts are used by noto-fonts-cjk
 2020-10-26 10:45:22 +08:00
Klemens Nanni
3216b85713 nixos/system-path: Add mkpasswd(1) 
Generating password hashes, e.g. when adding new users to the system
configuration, should work out-of-the-box and offline.
 2020-10-26 03:40:11 +01:00
Jonathan Ringer
37236c2a23 nixos/doc/rl-20.09: normalize highlights, reorder entries 2020-10-25 17:40:47 -07:00
WORLDofPEACE
4d71306596
Merge pull request #101516 from worldofpeace/gnome-polishing 
GNOME polishing from Q.A findings
 2020-10-25 18:41:34 -04:00
Andreas Rammhold
d4fb7daafd
nixos-build-vms: use the driverInteractive attribute instead 
This reverts commit aab534b894 & uses the
driverInteractive attribute for the test driver instead.

This has the same effect but removes the extra module in the
nixos-build-vms code.
 2020-10-25 20:14:53 +01:00
Andreas Rammhold
73635b859d
nixos/tests: fix runInMachine 
In 5500dc8 we introduced the --keep-vm-state flag and defaulted to that
flag not being set. This lead to the `runInMachine` tests not longer
working and that going unnoticed for quite some time now.
 2020-10-25 20:09:33 +01:00
Andreas Rammhold
fa25d84d13
nixos/tests: fix testDriver reference in runInMachine function 
In a previous commit I broke this as there is no longer one testDriver
but only a function to generate one based on some QEMU inputs.
 2020-10-25 20:09:33 +01:00
Andreas Rammhold
61b09f552c
nixos/tests: format the testing-python.nix file more consistenly 2020-10-25 20:09:33 +01:00
Andreas Rammhold
04100cd281
nixos/tests: restructure test driver so that QEMU is actually overriden 
Previously you would be able to override only the QEMU package to be
used in the test runner. Frankly that doesn't help a lot if you are
trying to get a graphical session. The graphical session requires the
option in the NixOS module system to bet set to the correct QEMU
package.

In this commit I moved most of the test node configuration and
transformations into the `mkDriver` function (previously called
`driver`). The motivation was to be able to create a `driver` instance
with a given QEMU package that will be used consistently througout the
test expression.
 2020-10-25 20:09:33 +01:00
Maximilian Bosch
a3041ab124
Merge pull request #101645 from andir/qemu-tests-fixup 
nixos/tests: only apply qemu parameters if the options are defined
 2020-10-25 19:25:50 +01:00
Konrad Borowski
254d30d4c9 test-driver.py: remove bufsize=1 from Popen calls 
According to Python documentation [0], `bufsize=1` is only meaningful in
text mode. As we don't pass in an argument called `universal_newlines`,
`encoding`, `errors` or `text` the file objects aren't opened in text
mode, which means the argument is ignored with a warning in Python 3.8.

    line buffering (buffering=1) isn't supported in binary mode,
    the default buffer size will be used

This commit removes this warning that appared when using
interactive test driver built with `-A driver`. This is done by
removing `bufsize=1` from Popen calls.

The default parameter when unspecified for `bufsize` is `-1` which
according to the documentation will be interpreted as
`io.DEFAULT_BUFFER_SIZE`. As mentioned by a warning, Python already
uses default buffer size when providing `buffering=1` parameter for
file objects not opened in text mode.

[0]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
 2020-10-25 16:22:07 +01:00
Benjamin Hipple
f98312fcb5
Merge pull request #79759 from lopsided98/syncoid-no-root 
nixos/syncoid: automatically setup privilege delegation
 2020-10-25 10:40:33 -04:00
Andreas Rammhold
f4d7493162
nixos/tests: only apply qemu parameters if the options are defined 
This fixes an eval error that occurred on hydra with the small channel
and the `nixos.tests.boot.biosCdrom.x86_64-linux` attribute:

> $ nix-instantiate nixos/release-small.nix -A nixos.tests.boot.biosCdrom.x86_64-linux
> warning: unknown setting 'experimental-features'
> error: The option `virtualisation.qemu' does not exist. Definition values:
> - In `/home/andi/dev/nixos/nixpkgs/nixos/modules/testing/test-instrumentation.nix':
>     {
>       consoles = [ ];
>       package = {
> 	_type = "override";
> 	content = <derivation /nix/store/q72h2cdcb9zjgiay5gdgzwddjkbjr7xq-qemu-host-cpu-only-for-vm-tests-5.1.0.drv>;
>     ...
> (use '--show-trace' to show detailed location information)

In bc2188b we changed test test-instrumentation to also set the QEMU
package that is being used. That change unfortunately caused us to
always assing values to the virtualisation.qemu.package option even when
the option is not defined. The original code was explicitly testing for
the consoles case but the then newly extended version did not adjust the
check as the intention was probably not clear.

With this commit we are always ensuring the entire virtualisation.qemu
section exists and can thus drop the individual tests for each of the
sections since the QEMU module always defines both the package and the
consoles option when it's root is defined..
 2020-10-25 13:42:01 +01:00
Scott Worley
f99b6369b1 nixos/tests/hadoop: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
c25ccf6b4b nixos/tests/docker-tools: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
f6ecfdac39 nixos/tests/bitcoind: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
0812bb843d nixos/tests/corerad: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
ad96a2e52b nixos/tests/cfssl: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
8adaa71b52 nixos/tests/caddy: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
ecbd2a8bc1 nixos/tests/cadvisor: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
eff7338d98 nixos/tests/convos: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
1abfb504b1 nixos/tests/hitch: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
19034ed7a4 nixos/tests/metabase: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
d16e547f2e nixos/tests/wordpress: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
6e46a88dfb nixos/tests/unit-php: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
726950775d nixos/tests/sympa: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
238dbb4517 nixos/tests/limesurvey: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
1d5130a97f nixos/tests/leaps: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
22f52cdb79 nixos/tests/osrm-backend: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
1332215d59 nixos/tests/php: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
8bf2635afc nixos/tests/oci-containers: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
7b51945dcb nixos/tests/trickster: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
ebeb8d7287 nixos/tests/sslh: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
8a9554bf9b nixos/tests/service-runner: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
a6a9e3188d nixos/tests/spacecookie: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
1959ab707c nixos/tests/victoriametrics: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
c69d4eda3d nixos/tests/uwsgi: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
ba0eda6cc5 nixos/tests/trezord: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
47fd1c5356 nixos/tests/morty: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
4fc64f27c7 nixos/tests/paperless: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
bbd1f02b16 nixos/tests/hound: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
bc4f47c19a nixos/tests/go-neb: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
d58ef9d20b nixos/tests/peerflix: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
7fbe33fde3 nixos/tests/nzbget: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
e9c61e813a nixos/tests/mailcatcher: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
e90e600ebb nixos/tests/upnp: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
416d9af81b nixos/tests/neo4j: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
6cd28e4f07 nixos/tests/mediawiki: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
0cb41a253f nixos/tests/trac: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
5e6dfb17f9 nixos/tests/haproxy: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
d2eec4bb11 nixos/tests/prometheus: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
a91aa489a6 nixos/tests/matrix-synapse: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
3313487997 nixos/tests/syncthing: Use curl --fail 2020-10-25 11:01:29 +01:00
Felix Tenley
542f75079b nixos/mosquitto: add passwordFile and hashedPasswordFile options 2020-10-25 10:53:38 +01:00
Gabriel Ebner
a8a018ddc0
Merge pull request #101409 from rycee/dbus-warning 2020-10-25 09:16:28 +01:00
Aaron Andersen
a160fa008d
Merge pull request #100063 from aanderse/nixos/powerdns 
nixos/powerdns: use upstream systemd unit
 2020-10-24 18:47:43 -04:00
Andreas Rammhold
250fb4611f
Merge pull request #100456 from maralorn/boolToString 
treewide: De-inline uses of lib.boolToString
 2020-10-25 00:45:11 +02:00
Robert Helgesson
94819fdb5f
nixos/dbus: re-add a dummy socketActivated option 
If set, then issue a warning instead of an error as previously.
 2020-10-24 23:01:00 +02:00
rnhmjoj
d5d6f619d4 nixosTests.powerdns: test a complete setup 
The test now check the following things:
- Configuring a MySQL server to hold the records
- Loading the PowerDNS schema from file
- Adding records through pdnsutil
 2020-10-24 16:41:34 -04:00
Aaron Andersen
6393835b8d nixos/powerdns: update release notes 2020-10-24 16:41:32 -04:00
Aaron Andersen
4f5d3794d3 nixos/powerdns: use upstream systemd unit 2020-10-24 16:40:20 -04:00
WORLDofPEACE
6bc94d149b
Merge pull request #101563 from worldofpeace/fix-pantheon-greeter-brightness 
nixos/lightdm: make lightdm user shell bash
 2020-10-24 11:56:34 -04:00
WORLDofPEACE
ef803ab1bb
Merge pull request #100199 from worldofpeace/seeded-config 
nixos/tools: add desktopConfiguration option (to seed configuration into configuration.nix)
 2020-10-24 11:35:33 -04:00
WORLDofPEACE
c134f6443a nixos/lightdm: make lightdm user shell bash 
In https://github.com/NixOS/nixpkgs/issues/100119 pantheon's greeter
has g-s-d running which allows brightness controls via pkexec.
This is changed in newer versions of g-s-d (pantheon uses a fork currently),
but whenever brightness is changed with a shell of `shadow` we get
```
Oct 10 23:51:44 kirXps pkexec[18722]: lightdm: Executing command [USER=root] [TTY=unknown] [CWD=/var/lib/lightdm] [COMMAND=/run/current-system/sw/bin/elementary-settings-daemon/gsd-backlight-helper /sys/devices/pci0000:00/0000:00:02.0/drm/card0/card0-eDP-1/intel_backlight 65587]
```

I'm not sure this should be strictly needed, so we should try to
revert later on when pantheon's g-s-d is updated.
 2020-10-24 11:28:18 -04:00
WORLDofPEACE
39d1599767 installation-cd-graphical-gnome: add firefox to favorite-apps 2020-10-24 11:14:44 -04:00
WORLDofPEACE
7df6af303e nixos/gnome3: add gnome-calendar to favorites 2020-10-24 11:14:41 -04:00
WORLDofPEACE
9cee7772e6 nixos/gnome3: add favoriteAppsOverride option 
Rather messy and only needed for the installation cd, so it's
an internal option.
 2020-10-24 11:14:22 -04:00
WORLDofPEACE
d89deddd5d nixos/flatpak: introduce guiPackages 
This adds basically an indirection to systemPackages
to automatically install an interface for flatpak for their respective
environments. e.g if I enable pantheon and flatpak you'll get appcenter,
and on gnome you'll see gnome-software.

https://github.com/NixOS/nixpkgs/issues/99648#issuecomment-706691174
 2020-10-24 11:14:02 -04:00
WORLDofPEACE
b1587f9e19 nixos/gnome3: don't ship gnome-software 
This serves no purpose without flatpak https://github.com/NixOS/nixpkgs/issues/99648#issuecomment-706691174
 2020-10-24 11:14:01 -04:00
Maximilian Bosch
48612c79b1
Merge pull request #101473 from Ma27/nixos-build-vms-qemu 
nixos/nixos-build-vms: use `pkgs.qemu` for virtualisation
 2020-10-24 14:43:52 +02:00
Fabian Möller
253954232e nixosTests.ferm: fix network timeout 
The subtests could start before the server has configured it's IP
addresses and therefore timeout.
 2020-10-23 18:12:45 -07:00
Fabian Möller
e83bd25aec nixosTests.certmgr: fix systemd test 
Nginx fails to start, because it can't read the certificate file. This
happens because PrivateTmp is set for the service, which makes the
system wide /tmp inaccessible.
 2020-10-23 18:09:50 -07:00
WORLDofPEACE
70dc25abd9 nixos/gnome3: don't put epiphany in favorite apps 2020-10-23 20:20:07 -04:00
Doron Behar
c90450014f
Merge pull request #101480 from Flakebi/salt 2020-10-24 01:31:41 +03:00
Jan Tojnar
3a73543401
Merge pull request #93725 from nglen/pipewire 2020-10-24 00:05:33 +02:00
WilliButz
993437d0d6
Merge pull request #96511 from Zopieux/rtl_433_prom 
Add rtl_433 Prometheus exporter
 2020-10-23 23:24:38 +02:00
Nathaniel Glen
57510bf522 pipewire: cleanup path testing 2020-10-23 16:35:48 -04:00
Alexandre Macabies
121bc17ab9 nixos/prometheus-rtl_433-exporter: new module 2020-10-23 20:33:42 +02:00
Nathaniel Glen
f6745d06f5 nixos/pipewire: cleanup module 2020-10-23 13:51:41 -04:00
Flakebi
e0ea4826f4
salt: remove aneeshusa and add Flakebi as maintainer 2020-10-23 19:48:48 +02:00
Doron Behar
649a0560e0
Merge pull request #101364 from doronbehar/doc/nixos/manual-contrib 
nixos/doc: Mention how to contribute to it
 2020-10-23 19:48:56 +03:00
WORLDofPEACE
b44e32988c
Merge pull request #101274 from worldofpeace/gnome-logout-button 
nixos/display-managers: add sessionData.desktops to XDG_DATA_DIRS
 2020-10-23 11:46:46 -04:00
Maximilian Bosch
aab534b894
nixos/nixos-build-vms: use pkgs.qemu for virtualisation 
When I test a change e.g. in the module system manually, I usually use
`nixos-build-vms(8)` which also gives me a QEMU window where I can play
around in the freshly built VM.

It seems as this has changed recently when the default package for
non-interactive VM tests using the same framework was switched to
`pkgs.qemu_test` to reduce the closure size. While this is a reasonable
decision for our CI tests, I think that you really want a QEMU window of
the VM by default when using `nixos-build-vms(8)`.

[1] bc2188b083
 2020-10-23 17:37:57 +02:00
Anton Plotnikov
1321ae850c
fido2luks: 0.2.3 -> 0.2.15 
Also remove interactive flag from initrd, because of broken io.
 2020-10-23 11:03:31 +03:00
Joe Edmonds
1d420c8115
nixos/ssmtp: minor typo fix 2020-10-22 09:34:21 -07:00
Doron Behar
7fb1e3af77 nixos/doc: Mention how to contribute to it 
Instead of putting a README in the directory where the manual is
written, put the information from it straight in the docs themselves.
It's a bit untrivial to guess the manual is located exactly there for
contributers.
 2020-10-22 17:40:48 +03:00
Lucas Savva
dad06fb922
nixos/tests/acme: Hard code test certificates 
The added README.md explains why this has been done.
 2020-10-22 14:06:19 +01:00
Lucas Savva
89d134b3fd
nixos/acme: Use more secure chmods 
Previous settings would make files executable in
the certs directories.
 2020-10-22 14:04:31 +01:00
Eelco Dolstra
d28565a1c6
nix: 2.3.7 -> 2.3.8 2020-10-22 14:47:55 +02:00
Maximilian Bosch
98170761a8
Merge pull request #101222 from omasanori/ssh-kex 
nixos/sshd: update kexAlgorithms, fix links
 2020-10-22 13:27:47 +02:00
Eelco Dolstra
05bdfd6f2f
Merge pull request #98973 from Ma27/bump-hydra 
hydra-unstable: 2020-09-02 -> 2020-10-20
 2020-10-22 12:01:13 +02:00
Andreas Rammhold
89351525fa
Merge pull request #101246 from rnhmjoj/vm-fix 
nixos: fix qemu_test being used in normal VMs
 2020-10-22 11:09:05 +02:00
symphorien
9e8eaea484
nixos/sslh: fix usage of the now removed ssl probe (#101087) 
and document
 2020-10-21 21:34:35 +02:00
WORLDofPEACE
755ba171c7 nixos/display-managers: add sessionData.desktops to XDG_DATA_DIRS 
Fixes #100108

Alternative to https://github.com/NixOS/nixpkgs/pull/100112 which doesn't break stuff.
 2020-10-21 14:39:39 -04:00
Maximilian Bosch
1308817e05
nixos/hydra: remove hydra-migration upgrade path 
This should NOT be backported to 20.09!

When 21.03 is released, the DB changes are about a year old and
operators had two release cycles for the upgrade. At this point it
should be fair to remove the compat layer to reduce the complexity of
the module itself.
 2020-10-21 18:03:04 +02:00
rnhmjoj
bc2188b083
nixos: fix qemu_test being used in normal VMs 
This is an attempt to fixup PR #49403.
 2020-10-21 16:38:04 +02:00
Izorkin
d59bfded58
nixos/dhcpcd: if disabled IPv6 don't solicit or accept IPv6 2020-10-21 14:56:08 +03:00
Martin Weinelt
c821e0d4be nixos/babeld: lock down service 
→ Overall exposure level for babeld.service: 2.2 OK 🙂
 2020-10-21 12:26:02 +02:00
Andreas Rammhold
f6cd17269e
Merge pull request #49403 from andir/qemu_test_reduce_closure 
qemu_test: disable features that are not needed for tests (closure 641 -> 335.3M)
 2020-10-21 00:41:01 +02:00
Masanori Ogino
8875db4976 nixos/sshd: update kexAlgorithms, fix links 
The `curve25519-sha256` key exchange method is defined in RFC 8731 that
is identical to curve25519-sha256@libssh.org. OpenSSH supports the
method since version 7.4, released on 2016-12-19. It is literally a
violation of the "both in Secure Secure Shell and Mozilla guidelines"
rule, but it provides essentially the same but a future-proof default.

Also, links to the Mozilla OpenSSH guidelines are updated to refer to
the current place.

Signed-off-by: Masanori Ogino <167209+omasanori@users.noreply.github.com>
 2020-10-21 07:39:50 +09:00
Florian Klink
6e5ccaa34f
Merge pull request #100657 from flokli/network-manager-sstp 
networkmanager-sstp: init at unstable-2020-04-20, bump sstp from 1.0.12 to 1.0.13
 2020-10-21 00:33:13 +02:00