paulgorman.org/technical

OpenBSD Mail Smarthost Runbook

(April 2018)

This is the day-to-day runbook for an OpenBSD mail smarthost. For installation and configuration, see https://paulgorman.org/technical/openbsd-smarthost.txt.html.

How do we update?

OpenBSD releases a new version every six months, along with upgrade instructions. OpenBSD provides updates for the current an previous version (i.e., for 12 months). The upgrade procedure is generally painless: boot from the install media, choose “upgrade”, and hit “y” a couple of times.

Between version upgrades, apply system updates like:

#  syspatch

Update add-on packages with:

#  pkg_add -Uu

How do we add/drop a user email address from the relay recipients list?

Add or delete the address from the text file /etc/mail/relay-recipients, then restart OpenSMTPD. (Is the restart necessary?)

$  doas vim /etc/mail/relay-recipients
$  doas rcctl restart smtpd

Where are the mail logs?

OpenSMTPD logs to /var/log/maillog.

By default, OpenBSD gzips the mail logs, which can still be accessed with tools like zgrep and zcat. However, we have set log rotation to not compress the mail logs. Older mail logs are available at /var/log/maillog.0, /var/log/maillog.1, etc.

How do we manage services? Where’s systemctl or service?

OpenBSD has its own init system (rc), with its own service management tool (rcctl).

$  rcctl help
usage:  rcctl get|getdef|set service | daemon [variable [arguments]]
        rcctl [-df] check|reload|restart|stop|start daemon ...
        rcctl disable|enable|order [daemon ...]
        rcctl ls all|failed|off|on|started|stopped
#  rcctl ls all
#  rcctl ls started
#  rcctl check smtpd
#  rcctl stop smtpd
#  rcctl start smtpd
#  rcctl restart smtpd
#  rcctl reload smtpd

What if the service fails to start?! Try starting it with the debug flag:

#  rcctl -d start amavisd

How do we add a spam score bonus or penalty (i.e., SpamAssassin soft white/blacklisting)?

Edit /etc/mail/amavis-senders. Each line in the file adds to or subtracts from the final spam score.

bad.example.com 10
good.example.org -5

After saving /etc/mail/amavis-senders:

$  doas rcctl restart spamassassin amavisd

(Is the service restart necessary? I’m not sure.)

Note: we specify the white/blacklist file in /etc/amavisd.conf:

@score_sender_maps = ({
	'.' => [
		read_hash("/etc/mail/amavis-senders"),
		…

A user is waiting more mail. Is it greylisted by spamd?

$  spamdb | grep my.user

What are we blocking with spamd?

A root cronjob updates the lists every hour. Since spamd works through PF, we can check the current firewall rules.

#  pfctl -s Tables
#  pfctl -t spamd-white -T show

What about the greylist?

Spamd maintains its greylist in /var/db/spamdb, which we can examine with the spamdb tool.

#  spamdb

(Is spamdb only for greylisted addresses, or for everything? If so, spamdb may be the better tool to examine and manipulate entries than pfctl.)

A large provider sends from a large pool of mail senders, so the greylist triplets never match and never get whitelisted!

Add the domain to /etc/mail/common_domains, like:

amazonses.com
aol.com
att.net
charter.net
comcast.net
cox.net
email.com
facebook.com
facebookmail.com
gmail.com
googlemail.com
google.com
hotmail.com
icloud.com
mac.com
me.com
mail.com
msn.com
live.com
outlook.com
sbcglobal.net
verizon.net
yahoo.com

Use the spf_update_pf utility script to grab the SPF records for those domains, and insert them into our whitelist.

$  doas /usr/local/bin/spf_update_pf
40 addresses added.
$  doas rcctl restart smtpd

https://github.com/akpoff/spf_fetch

After shutting down OpenBSD, its hypervisor still sees it as “running”!

Use shutdown -h -p +1. The -p flag allows power-down after halting the system.

Note that it’s safe to virsh destroy myvm a halted VM that was shut down without the -p flag.

What messages are currently in the OpenSMTPD queue?

#  smtpctl show queue

How do we manage the Amavis quarantine?

Amavis quarantines bad messages in /var/virusmails. When it quarantines a message, it leaves an entry in the mail log with the quarantine file name (e.g., quarantine: badh-zROkaXVT9p33). Release the message from quarantine, and deliver it to its recipient, like:

#  amavisd-release badh-zROkaXVT9p33

What do these obscure SpamAssassin tags mean (e.g., RCVD_IN_IADB_DOPTIN)?

Find the best, though also terse, description in the SpamAssassin code:

$  grep -i describe /usr/local/share/spamassassin/* | grep RCVD_IN_IADB_DOPTIN

What is doas? Where’s sudo?

Although the OpenBSD project originally developed sudo, they’ve come to view it as too complex for how most people use it. OpenBSD has developed doas as simpler replacement for sudo. If typing doas is bothersome, create a shell alias, like:

alias sudo='doas'

See doas(1).

There are still use cases for sudo. It’s available as a package, if needed.

The shell is weird.

OpenBSD defaults to ksh. It’s not too bad once you get used to it.

Check your current shell with:

$  echo $SHELL

The bash shell is available as a package, along with zsh and others. Check the available shells with:

$  cat /etc/shells

If Bash is not listed, install it like:

#  pkg_add bash

Once Bash is installed, change the shell for your user like:

$  chsh -s /usr/local/bin/bash

How do we add a new user? How do we add a user to a group?

The adduser and rmuser prompt for details about the creation or removal of a user. The adduser command is generally what we want, rather than the lower-level useradd command.

#  adduser

Drop a user with the userdel command.

We generally add IT staff to the “wheel” group, which allows them to use doas. Add an existing user to the wheel group:

#  usermod -G wheel alice

Change passwords with the passwd command.

Where is gtimeout (or another GNU utility)?

The coreutils package provides many GNU utilities. Many of the binaries from this package are named like gtimeout. See a list of binaries in the package:

$  pkg_info -L coreutils | grep '/bin/'

How do we tweak a spam score based on message headers, like the Subject line?

Edit /etc/mail/spamassassin/local.cf, and add a rule like:

header INVOICE_PHISH_SUBJECT Subject =~ /\binvoice/i
score INVOICE_PHISH_SUBJECT 2

Then:

$  doas rcctl restart spamassassin amavisd
spamassassin(ok)
spamassassin(ok)
amavisd(ok)
amavisd(ok)

A message is stuck or bouncing around in the mail queue!

$  smtpctl pause mta
$  smtpctl show queue
$  smtpctl envelope a9de9a39de27101b
$  smtpctl remove a9de9a39de27101b
$  smtpctl resume mta

Is the soft white/blacklist (i.e., score_sender_maps) from Amavis working?!

Matching messages get tagged. Grep the mail logs for AM.WBL.

The output of spamdb is hard to read!

#!/usr/bin/env perl

# This script formats the output of `spamdb` on OpenBSD to make it more readable by humans.
# Paul Gorman, May 2018

use strict;
use warnings;
use POSIX qw(strftime);

sub hd {
	return strftime("%H:%M:%S %b %d", localtime($_[0]));
}

my @spamdb = split(/\n/, `spamdb`);

foreach my $line (@spamdb) {
	my @l = split(/\|/, $line);
	if ($l[0] eq 'GREY') {
		my $ip = $l[1];
		my $helo = $l[2];
		my $sadr = substr($l[3], 1, -1);
		my $radr = substr($l[4], 1, -1);
		my $fdt = strftime("%b %d %H:%M:%S", localtime($l[5]));
		my $pdt = strftime("%b %d %H:%M:%S", localtime($l[6]));
		my $edt = strftime("%b %d %H:%M:%S", localtime($l[7]));
		my $bct = $l[8];
		my $pct = $l[9];
		printf("%s  1ST %s  PASS %s  EXP %s    BLCT %-3s PSCT %-3s    %-15s %-30s %s -> %s\n",
		   $l[0],   $fdt,    $pdt,   $edt,      $bct,     $pct, $ip, $helo, $sadr, $radr);
	} elsif ($l[0] eq 'WHITE') {
		my $ip = $l[1];
		my $fdt = strftime("%b %d %H:%M:%S", localtime($l[4]));
		my $pdt = strftime("%b %d %H:%M:%S", localtime($l[5]));
		my $edt = strftime("%b %d %H:%M:%S", localtime($l[6]));
		my $bct = $l[7];
		my $pct = $l[8];
		printf("%s 1ST %s  PASS %s  EXP %s    BLCT %-3s PSCT %-3s    %s \n",
			$l[0],  $fdt,   $pdt,   $edt,      $bct,      $pct,   $ip);
	} else {
		print join('    ', @l), "\n";
	}
}

What spam scores have passed messages received?

To sort output of the following script by score:

mlog | sort -n -k 10

Here’s the mlog script:

#!/usr/bin/env perl

# This script get Passed lines from the mail log (Amavis+OpenSMTPD), and pretty prints them.
# Paul Gorman, May 2018

use strict;
use warnings;

# May 11 12:30:44 nostromo amavis[96961]: (96961-08) Passed CLEAN {RelayedInbound}, [127.0.0.1] [216.82.180.36] /ESMTP <prvs=6625f3126=john.smith@example.com> -> <Kristen.mozol@example.com>, (ESMTPS://216.82.180.36), Message-ID: <FF68A08914FEC846A95D6980E128406D0110B89EC9@s1flokydce2kx06.dm0001.info53.com>, mail_id: 9zW_7ljoGzGW, b: yTqus-1QJ, Hits: -5.11, size: 5213275, queued_as: 250 2.0.0: 7433ece1 Message accepted for delivery, Subject: "Mail float", From: <Aaron.Davidson@53.com> (dkim:AUTHOR), helo=, Tests: [DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,HTML_MESSAGE=0.001,RCVD_IN_DNSWL_HI=-5,SPF_PASS=-0.001,T_RP_MATCHES_RCVD=-0.01], autolearn=ham autolearn_force=no, autolearnscore=-5.109, dkim_i=Aaron.Davidson@53.com,@53.com, dkim_sd=main:53.com, 7886 ms

# Regex captures:
# 1 sender ip
# 2 from email
# 3 to email
# 4 hits
# 5 size
# 6 subject

my $r = '.+amavis.+Passed.+\[127.0.0.1\] \[(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\].+/ESMTP <(.+@.+)> -> <(.+@.+)>, \(ESMTP.+Hits:\s+([\-\d\.]+), size: (\d+),.+Subject: "(.+)", From:.+';

my @log = split(/\n/, `cat /var/log/maillog`);

foreach my $line (@log) {
	if ($line =~ /$r/) {
		my @l = split(/\s+/, $line);
		my $s = reverse(substr(reverse($2), 0, 50));
		printf("%s %2s %s  %9s B  %-15s  %50.50s -> %-40.40s  %7.3f hits  \"%.65s\"\n",
			$l[0], $l[1], $l[2], $5, $1, $s,    $3,       $4,           $6);
	}
}

References

mail flow diagram