On spectrum-os.org, mailman-web is run at /lists. With this change,
it's possible for us to switch from a custom uWSGI configuration to
the one now built in to the Mailman module.
When switching between different NixOS configurations (with and
without nullmailer and other services), it can happen that the UID of
the nullmailer user changes. When it happens, the nullmailer service
happily starts, but the user cannot send any email, because the
sendmail wrapper doesn't have permission to write them to the queue.
This commit prevents that. Instead of creating the directories by the
nullmailer user, which doesn't have permissions to change ownership,
we now create them by the systemd-tmpfiles, which has sufficient
permissions to adjust ownership.
most of these are hidden because they're either part of a submodule that
doesn't have its type rendered (eg because the submodule type is used in
an either type) or because they are explicitly hidden. some of them are
merely hidden from nix-doc-munge by how their option is put together.
conversions were done using https://github.com/pennae/nix-doc-munge
using (probably) rev f34e145 running
nix-doc-munge nixos/**/*.nix
nix-doc-munge --import nixos/**/*.nix
the tool ensures that only changes that could affect the generated
manual *but don't* are committed, other changes require manual review
and are discarded.
this mostly means marking options that use markdown already
appropriately and making a few adjustments so they still render
correctly. notable for nftables we have to transform the md links
because the manpage would not render them correctly otherwise.
using regular strings works well for docbook because docbook is not as
whitespace-sensitive as markdown. markdown would render all of these as
code blocks when given the chance.
now nix-doc-munge will not introduce whitespace changes when it replaces
manpage references with the MD equivalent.
no change to the manpage, changes to the HTML manual are whitespace only.
make (almost) all links appear on only a single line, with no
unnecessary whitespace, using double quotes for attributes. this lets us
automatically convert them to markdown easily.
the few remaining links are extremely long link in a gnome module, we'll
come back to those at a later date.
we can't embed syntactic annotations of this kind in markdown code
blocks without yet another extension. replaceable is rare enough to make
this not much worth it, so we'll go with «thing» instead. the module
system already uses this format for its placeholder names in attrsOf
paths.
markdown can't represent the difference without another extension and
both the html manual and the manpage render them the same, so keeping the
distinction is not very useful on its own. with the distinction removed
we can automatically convert many options that use <code> tags to markdown.
the manpage remains unchanged, html manual does not render
differently (but class names on code tags do change from "code" to "literal").
the conversion procedure is simple:
- find all things that look like options, ie calls to either `mkOption`
or `lib.mkOption` that take an attrset. remember the attrset as the
option
- for all options, find a `description` attribute who's value is not a
call to `mdDoc` or `lib.mdDoc`
- textually convert the entire value of the attribute to MD with a few
simple regexes (the set from mdize-module.sh)
- if the change produced a change in the manual output, discard
- if the change kept the manual unchanged, add some text to the
description to make sure we've actually found an option. if the
manual changes this time, keep the converted description
this procedure converts 80% of nixos options to markdown. around 2000
options remain to be inspected, but most of those fail the "does not
change the manual output check": currently the MD conversion process
does not faithfully convert docbook tags like <code> and <package>, so
any option using such tags will not be converted at all.
Otherwise, it wouldn't get restarted when a new system configuration
was activatad, so the Postfix configuration wouldn't be updated.
Fixes: fb2fa1b50f ("nixos/postfix: pull setup into its own unit")
* Removed unused `.package`-option.
* Added explicit postgresql support.
* Create a new meta-package for mailman to make sure each component has
the **same** python and packages can be downgraded if needed (e.g.
psycopg2 or sqlalchemy) without interfering with `pythonPackages` in any way.
* Document why certain python overrides are needed.
Closes#170035Closes#158424
Add support for enabling confinement
but does not enable it by default yet
because so far no module within NixOS uses confinement
hence that would set a precedent.
On first run, Postfix will refuse to start if it's started before
Mailman is up, because it'll try to read the map files generated
Mailman the first time it's started, and they won't exist yet. To fix
this, make sure Postfix isn't started until after Mailman is up if
they're both activated at the same time.
Consider a service that generates postfix lookup tables with
postmap(1), like Mailman. It needs the Postfix configuration file to
exist, but Postfix qmgr needs all the lookup tables its configured
with to exist before it starts. So the service that runs postmap
needs to run after the Postfix configuration and directory structure
is generated, but before Postfix itself is started. To enable this,
we split Postfix into two units: a oneshot unit that sets up the
configuration, and a longrun unit that supervises the Postfix
daemons. The postmap services can then be inserted in between these
two units.
* nixos/opensmtpd: Add missing brackets in config
Without this commit, you end up missing the sendmail suid wrapper,
because the "program" attribute would not override the right thing.
* Update nixos/modules/services/mail/opensmtpd.nix
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
Nullmailer expects that this directory exists (see
073f4e9c5d/doc/nullmailer-send.8 (L185)).
When it doesn't and an email cannot be sent due to a permanent failure
or has been in the queue longer than queuelifetime (7 days), message
"Can't rename file: No such file or directory" starts appearing in the
log and nullmailer never sends "Could not send message" notification.
This means that the user may never learn that his email was not
delivered.
Without this change, mailman-settings.service is not guaranteed to
complete before dependent services. This can lead to various errors
like:
mailman-web-setup.service: Changing to the requested working directory failed: No such file or directory
Postfix has started outputting an error on startup that it can't parse
the compatibility level 9999.
Instead, just set the compatibility level to be identical to the current
version, which seems to be the (new) intent for the compatibility level.
An empty list results in no CapabilityBoundingSet at all, an empty
string however will set `CapabilityBoundingSet=`, which represents a
closed set.
Related: #120617
An empty list results in no CapabilityBoundingSet at all, an empty
string however will set `CapabilityBoundingSet=`, which represents a
closed set.
Related: #120617
With the config suggested in the module docs both Mailman core and
Hyperkitty are running, but Mailman core can not connect to Hyperkitty,
since the default hyperkitty.baseUrl is not set up by the module.
This adds a http listener to the uwsgi config and changes the default
hyperkitty.baseUrl to connect to this http listener.
* Make it clearer what code comments apply to
* Fix the state directory (this was changed in the update)
* Add m1cr0man as a maintaner
Co-authored-by: Lucas Savva <lucas@m1cr0man.com>
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
The NixOS 21.03 release has been delayed to 21.05. See NixOS/rfcs#80.
There are two instances of 21.03 which have been left as is, since they
are in stateVersion comparisons. This will ensure that existing user
configurations which refer to 21.03 will continue to work.
For sa-update we care about two successful codes:
* 1 -> no updates available: exit successfully
* 0 -> updates have been installed: run sa-compile and pass
through its return code
sa-compile speeds up processing the rules by compiling them from Perl to
C. This needs to be run after every update and is saved in the local
state directory by Perl and SpamAssassin version.
Let systemd create SpamAssassin's state directory and populate it using the
regular updater service. Depend on the updater service on boot but do not
propagate failure to the main service.
spamd's commands to start and reload the service are still executed as
root but user/group are set to properly chown the state directory to the
target user. spamd drops privileges itself for its runner children but
preserves root on the main daemon (to listen and re-exec).
sa-update currently runs as part of the pre-start script of spamd. The
network is not guaranteed to be online at that point and even if we
were to depend on that, it makes the bootup brittle, as there is a
reliance on SpamAssassin's update server as a startup dependency on
boot.
Refactor the setup to move the pre-start script into its own unit.
This allows to perform the setup task only once. Continuous updates
are already done by sa-update.service triggered by sa-update.timer.
Only run sa-update in case /var/lib/spamassassin is empty.
While we are on it, let sa-update.service depend on the network being
online.
Now that smtp_tls_security_level is using mkDefault, and therefore can
be overridden, there's no need for an option for overriding it to a
specific value.
I run Postfix on my workstation as a smarthost, where it only ever
talks to my SMTP server. Because I know it'll only ever connect to
this server, and because I know this server supports TLS, I'd like to
set smtp_tls_security_level to "encrypt" so Postfix won't fall back to
an unencrypted connection.
b478e0043c removed every other instance of uwsgi.service and
httpd.service from Mailman before and requiredBy lists. This one
looks like it was just missed, since I can't see a reason to keep it
but remove the others.
Fixes: b478e0043c ("nixos/mailman: refactor")
Exim spawns a new queue runner every n minutes as configured by the
argument to -q; up to queue_run_max can be active at the same time.
Spawning a queue runner only every 30 mins means that a message that
failed delivery on the first attempt (e.g. due to greylisting) will only
be retried 30 minutes later.
A queue runner will immediately exit if the queue is empty, so it is
more a function on how quickly Exim will scale to mail load and how
quickly it will retry than something that is taxing on an otherwise
empty system.
Mailman can now work with MTAs other than Postfix. You'll have to configure
it yourself using the options in `services.mailman.settings.mta`.
This addition is reflected in the release notes for 21.03.
These were broken since 2016:
f0367da7d1
since StartLimitIntervalSec got moved into [Unit] from [Service].
StartLimitBurst has also been moved accordingly, so let's fix that one
too.
NixOS systems have been producing logs such as:
/nix/store/wf98r55aszi1bkmln1lvdbp7znsfr70i-unit-caddy.service/caddy.service:31:
Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
I have also removed some unnecessary duplication in units disabling
rate limiting since setting either interval or burst to zero disables it
(ad16158c10/src/basic/ratelimit.c (L16))
`sslCACert` was used for trust store of client and server certificates. Since `smtpd_tls_ask_ccert` defaults to no the setup of `smtpd_tls_CApath` was removed.
>By default (see smtpd_tls_ask_ccert), client certificates are not requested, and smtpd_tls_CApath should remain empty.
see http://www.postfix.org/postconf.5.html#smtpd_tls_CAfile
- Add serve.enable option, which configures uwsgi and nginx to serve
the mailman-web application;
- Configure services to log to the journal, where possible. Mailman
Core does not provide any options for this, but will now log to
/var/log/mailman;
- Use a unified python environment for all components, with an
extraPackages option to allow use of postgres support and similar;
- Configure mailman's postfix module such that it can generate the
domain and lmtp maps;
- Fix formatting for option examples;
- Provide a mailman-web user to run the uwsgi service by default
- Refactor Hyperkitty's periodic jobs to reduce repetition in the
expressions;
- Remove service dependencies not related to functionality included in
the module, such as httpd -- these should be configured in user config
when used;
- Move static files root to /var/lib/mailman-web-static by default. This avoids
permission issues when a static file web server attempts to access
/var/lib/mailman which is private to mailman. The location can still
be changed by setting services.mailman.webSettings.STATIC_ROOT;
- Remove the webRoot option, which seems to have been included by
accident, being an unsuitable directory for serving via HTTP.
- Rename mailman-web.service to mailman-web-setup.service, since it
doesn't actually serve mailman-web. There is now a
mailman-uwsgi.service if serve.enable is set to true.
Specifying mailboxes as a list isn't a good approach since this makes it
impossible to override values. For backwards-compatibility, it's still
possible to declare a list of mailboxes, but a deprecation warning will
be shown.
The setgid is currently required for offline enqueuing, and
unfortunately smtpctl is currently not split from sendmail so there's
little running around it.
Rework withExtensions / buildEnv to handle currently enabled
extensions better and make them compatible with override. They now
accept a function with the named arguments enabled and all, where
enabled is a list of currently enabled extensions and all is the set
of all extensions. This gives us several nice properties:
- You always get the right version of the list of currently enabled
extensions
- Invocations chain
- It works well with overridden PHP packages - you always get the
correct versions of extensions
As a contrived example of what's possible, you can add ImageMagick,
then override the version and disable fpm, then disable cgi, and
lastly remove the zip extension like this:
{ pkgs ? (import <nixpkgs>) {} }:
with pkgs;
let
phpWithImagick = php74.withExtensions ({ all, enabled }: enabled ++ [ all.imagick ]);
phpWithImagickWithoutFpm743 = phpWithImagick.override {
version = "7.4.3";
sha256 = "wVF7pJV4+y3MZMc6Ptx21PxQfEp6xjmYFYTMfTtMbRQ=";
fpmSupport = false;
};
phpWithImagickWithoutFpmZip743 = phpWithImagickWithoutFpm743.withExtensions (
{ enabled, all }:
lib.filter (e: e != all.zip) enabled);
phpWithImagickWithoutFpmZipCgi743 = phpWithImagickWithoutFpmZip743.override {
cgiSupport = false;
};
in
phpWithImagickWithoutFpmZipCgi743
Prior to this fix, changes to certain settings would not be applied
automatically and users would have to know to manually restart the
affected service. A prime example of this is
`services.mailman.hyperkitty.baseUrl`, or various things that affect
`mailman3/settings.py`
Previously, some files were copied into the Nixpkgs tree, which meant
we wouldn't easily be able to update them, and was also just messy.
The reason it was done that way before was so that a few NixOS
options could be substituted in. Some problems with doing it this way
were that the _package_ changed depending on the values of the
settings, which is pretty strange, and also that it only allowed those
few settings to be set.
In the new model, mailman-web is a usable package without needing to
override, and I've implemented the NixOS options in a much more
flexible way. NixOS' mailman-web config file first reads the
mailman-web settings to use as defaults, but then it loads another
configuration file generated from the new services.mailman.webSettings
option, so _any_ mailman-web Django setting can be customised by the
user, rather than just the three that were supported before. I've
kept the old options, but there might not really be any good reason to
keep them.
We already had python3Packages.mailman, but that's only really usable
as a library. The only other option was to create a whole Python
environment, which was undesirable to install as a system-wide
package.
It's likely that a user might want to set multiple values for
relay_domains, transport_maps, and local_recipient_maps, and the order
is significant. This means that there's no good way to set these
across multiple NixOS modules, and they should probably all be set
together in the user's Postfix configuration.
So, rather than setting these in the Mailman module, just make the
Mailman module check that the values it needs to occur somewhere, and
advise the user on what to set if not.
This replaces all Mailman secrets with ones that are generated the
first time the service is run. This replaces the hyperkittyApiKey
option, which would lead to a secret in the world-readable store.
Even worse were the secrets hard-coded into mailman-web, which are not
just world-readable, but identical for all users!
services.mailman.hyperkittyApiKey has been removed, and so can no
longer be used to determine whether to enable Hyperkitty. In its
place, there is a new option, services.mailman.hyperkitty.enable. For
consistency, services.mailman.hyperkittyBaseUrl has been renamed to
services.mailman.hyperkitty.baseUrl.
Using a custom path in the Nix store meant that users of the module
couldn't add their own config files, which is a desirable feature. I
don't think avoiding /etc buys us anything.
Motivation:
if enableQuota is true, mail plugins cannot be enabled in extraConfig
because of the problem described here:
https://doc.dovecot.org/configuration_manual/config_file/config_file_syntax/#variable-expansion
doveconf: Warning: /etc/dovecot/dovecot.conf line 8: Global setting
mail_plugins won't change the setting inside an earlier filter at
/etc/dovecot/dovecot.conf line 5 (if this is intentional, avoid this
warning by moving the global setting before /etc/dovecot/dovecot.conf
line 5)
The php installer creates a random one, but we bypass it, so we have
to create one ourselves.
This should be backward compatible as encryption is used for session
cookies only: users at the time of the upgrade will be logged out but
nothing more.
259b7fa065/config/config.inc.php.sample (L73)
If the database is local, use postgres peer authentication.
Otherwise, use a password file.
Leave database initialisation to postgresql.ensure*.
Leave /var/lib/roundcube creation to systemd.
Run php upgrade script as unpriviledged user.
Supporting a path here is important because it allows e.g. fetching a
configuration from a URL. To do this and provide the configuration as
a string, IFD would be necessary. It's just written into a path
anyway.
A centralized list for these renames is not good because:
- It breaks disabledModules for modules that have a rename defined
- Adding/removing renames for a module means having to find them in the
central file
- Merge conflicts due to multiple people editing the central file
When mailman-web restarts, it removes the generated "static" directory. This
breaks a currently running httpd process, which needs a re-start, too, to
obtain a new handle for the newly generated path.
Any system uid will do, so we let the system allocate
one for us. The 'mailman' group is gone entirely since
we don't need it. Users who wish to run the 'mailman'
administration utility can do so via 'sudo':
$ sudo -u mailman mailman info
Also, simplify the syntax of our user.users entry to
rely on an attribute set rather than a list.
The `keys.target` is used to indicate whether all NixOps keys were
successfully uploaded on an unattended reboot. However this can cause
startup issues e.g. with NixOS containers (see #67265) and can block
boots even though this might not be needed (e.g. with a dovecot2
instance running that doesn't need any of the NixOps keys).
As described in the NixOps manual[1], dependencies to keys should be
defined like this now:
``` nix
{
systemd.services.myservice = {
after = [ "secret-key.service" ];
wants = [ "secret-key.service" ];
};
}
```
However I'd leave the issue open until it's discussed whether or not to
keep `keys.target` in `nixpkgs`.
[1] https://nixos.org/nixops/manual/#idm140737322342384
The new option services.postfix.localRecipients allows
configuring the postfix option 'local_recipient_maps'. When
set to a list of user names (or patterns), that map
effectively replaces the lookup in the system's user
database that's used by default to determine which local
users are valid.
This option is useful to explicitly set local users that are
allowed to receive e-mail from the outside world. For local
injection i.e. via the 'sendmail' command this option has no
effect.
When using a different database, the evaluation fails as
`config.services.postgresql.package` is only set if `services.postgresql` is enabled.
Also, the systemd service shouldn't have a relation to postgres if a
remote database is used.
With this option it's possible to specify a custom expression for
`roundcube`, i.e. a roundcube environment with third-party plugins as
shown in the testcase.
When reworking the rspamd workers I disallowed `proxy` as a type and
instead used `rspamd_proxy` which is the correct name for that worker
type. That change breaks peoples existing config and so I have made this
commit which allows `proxy` as a worker type again but makes it behave
as `rspamd_proxy` and prints a warning if you use it.
The `rmilter` module has options for configuring `postfix` to use it but
since that module is deprecated because rspamd now has a builtin worker
that supports the milter protocol this commit adds similar `postfix`
integration options directly to the `rspamd` module.
