Fixes#241482.
Also fix test putting cluster in unhealthy `POOL_APP_NOT_ENABLED` state;
this seems to be the default state with Ceph 18.2.1 at least,
and it does not hurt to fix it now already in the way the Ceph docs say.
Also revert "nixosTests.ceph-single-node: remove dashboard check"
This reverts commit 41b27d7f4b.
Due to an [issue](https://www.spinics.net/lists/ceph-users/msg77812.html) with the cryptography python library, Ceph Dashboard and other mgr modules are currently broken, which will cause this test to always fail. Removing the check resolves this issue, and brings the test in line wit^Cthe other Ceph tests, which do not contain the dashboard check.
the default hasn't been changed since 2009
this can improve our test performances
nixos/tests: remove explicit memorySize <1024
1024MiB is now the default
This prevents services to be started before they're initialized, and
renders the `systemd.targets.ceph.wantedBy = lib.mkForce [];` hack in
the vm tests obsolete - The config now starts up ceph after a reboot,
too.
Let's take advantage of that, crash all VMs, and boot them up again.
Don't pass user and group to ceph, and rely on it to drop ceps, but let
systemd handle running it as the appropriate user.
This also inlines the extraServiceConfig into the makeService function,
as we have conditionals depending on daemonType there anyways.
Use StateDirectory to create directories in
/var/lib/ceph/${daemonType}/${clusterName}-${daemonId}.
There previously was a condition on daemonType being one of mds,mon,rgw
or mgr. We only instantiate makeServices with these types, and "osd" was
special.
In the osd case, test examples suggest it'd be in something like
/var/lib/ceph/osd/ceph-${cfg.osd0.name} - so it's not special at all,
but exactly like the pattern for the others.
During initialization, we also need these folders, before the unit is
started up. Move the mkdir -p commands in the vm tests to the line
immediately before they're required.
Rename the old ceph test to ceph-single-node and add a new test
ceph-multi-node. The ceph-single-node represents a dev cluster whereas
ceph-multi-node is closer to a prod cluster.
