The Arda.Homeunix.Net Mail Setup

Table of contents

1. Introduction
2. Preliminaries
3. The MTA (Mail Transfer Agent)
   1. Control Files
   2. The Vmail System Account
   3. .qmail Files
   4. System Aliases
   5. The User Database
   6. Run Scripts
4. qmail as a Null Client
   1. Control Files
   2. Run Scripts
5. The IMAP Server
   1. Configuration Options
   2. Mail Accounts
   3. Configuration Files
   4. Starting Courier
6. The Webmail Interface
7. Retrieving Mail from my ISP
8. Starting Processes with Daemontools
   1. Daemontools on Callisto
   2. Daemontools on Europa
9. Stopping Spam
   1. Spfilter
   2. TMDA (Tagged Message Delivery Agent)
10. Mail & DNS
11. Software Home Sites
12. Further Reading

You will find an overview of the Arda Network, including a very nice diagram, here.

All mail accounts for the domain reside on Callisto and all are virtual, managed by a single system account. Mail received by and generated on Metis, Europa, Io, and Thebe is forwarded to Callisto for delivery. Ganymede, being a Windows box, doesn’t have much to do with any of this except in the capacity of making sure Windows mail clients still work after I’ve changed something.

I’ve set up my mail server with the idea of being able to host multiple virtual domains for email accounts. People familiar with using multiple virtual domains will likely know about two very good packages designed to make virtual domains easier to use; vmailmgr and vpopmail.

I use neither of these packages in my setup, having opted for my own way of managing virtual domains. Because of my particular implementation, I don’t have access to many of the useful tools available in the two packages mentioned. This isn’t a big limitation for me because I need to deal with only a small number of virtual domains. For anyone contemplating setting up a mail server that will host a large number of virtual domains (I leave the interpretation of the word 'large' to the reader), I would recommend investigating either of the two packages mentioned.

Here is a list of the various software packages that I will be discussing in this document.

Software	Installed On
netqmail 1.05	Callisto Europa Io Metis Thebe
Courier IMAP 3.0.4	Callisto
Courierpassd 1.0.1	Callisto
Courierpasswd 1.0.1	Callisto
Courieruserinfo 1.0.1	Callisto
daemontools 0.76	Callisto Europa Io Metis Thebe
Fetchmail 6.2.5	Callisto
SquirrelMail 1.4.3a	Io
spfilter 0.59	Callisto
TMDA 1.0.3	Callisto
ucspi-tcp 0.88	Callisto

The MTA (Mail Transfer Agent)

I use qmail as my MTA for handling mail delivery for my domain. Qmail is installed on all my machines save Ganymede. In this section, I will describe qmail as it is installed on Callisto. See the section Qmail as a Null Client for a description of qmail installed on Europa. I also use the ucspi-tcp and daemontools packages to control and enhance the functionality of qmail.

There are numerous MTAs available nowadays and you might be asking yourself why I chose qmail over the others. My needs likely could be filled by any of the more common MTAs but there were a few considerations that made qmail win out over the others.

I did a bit of research and found that qmail was a little more mature than most of the more modern MTAs. I include Courier, Cyrus, Exim, and Postfix in this category. I also found numerous patches and add-ons to extend qmail’s functionality. From this, it looked like qmail was a stable, proven MTA that would have the features that I would need and that had a substantial developer community contributing to it. As it turns out, I am happy with the choice I made. Qmail has done everything I’ve needed it to do so far and has not caused me any trouble.

I’m not actually using stock qmail on any of my machines anymore. I’ve upgraded them all to use netqmail. For those who don’t know, netqmail is stock qmail distributed with a small number of patches and a script to apply the patches. You will find a link to netqmail in the Software Home Sites section. Once netqmail is installed, you configure it in the same way as stock qmail so this document still applies to people using qmail.

In addition to the patches included with netqmail, I typically add two more patches before installing it. Here are the patches I apply.

Patch	Description
starttls-2way-auth-20040725.patch	Provides startTLS and authentication capability to SMTP sessions
qregex-20040725.patch	Adds full regular expression matching to the badmailfrom control file and adds the badhelo and badmailto control files.

Because the process of building qmail was somewhat involved, I’ll explain it in depth.

1. First I unpacked the netqmail package and ran the collate.sh script to apply the included patches.
2. I then cd'ed into the resulting netqmail-1.05 directory and applied the first patch using the command; 'patch < /path/to/starttls-2way-auth-20040725.patch'. This patch applied cleanly.
3. I then applied the qregex patch with this command; 'patch < /path/to/qregex-20040725.patch'. This patch didn’t apply cleanly because it collided with some changes made by the first one. I used the *.rej files generated by patch to find the parts that failed and applied them manually.
4. Still in the netqmail-1.05 directory, I ran 'make setup check' to build and install netqmail.
5. Once installed, I set to work configuring netqmail.

You can find a description of how I used to install qmail from the FreeBSD ports tree here

I applied the starttls-2way-auth patch for two reasons:

1. I wanted to use authentication between mail clients and qmail-smtpd to restrict who could relay mail through my mail server. This method seemed more flexible and reliable than using host-based filtering.
   
   
2. Because usernames and passwords would be passed between mail clients and my server, I wanted to protect those passwords from network sniffers.

The starttls-2way-auth patch filled both these needs. This patch requires some additional configuration of qmail over a stock installation and I’ll go into these differences where applicable.

It is also important to remember to disable CRAM type authentication in qmail-smtpd if you aren’t going to be using it. To do this, undefine CRAM_MD5 in qmail-smtpd.c after applying the above patches and before building qmail. I didn’t at first and later found that qmail would advertise CRAM-MD5 authentication even though my mail server as configured doesn’t support it. This confused some other programs I have that interact with qmail and would cause them to fail authentication.

The starttls-2way-auth patch is actually a combination of three patches I found at www.qmail.org; qmail-remote_authenticated_smtp, qmail-smtpd-auth-043, and netqmail-1.05-tls-20040419. While qmail-smtpd-auth-043 can be used to tell qmail-smtpd to require authentication before accepting mail, qmail-remote_authenticated_smtp can tell qmail-remote to use authentication when sending mail. I’m not using this later capability yet, but I thought I might, so I included it in my qmail setup.

I’m using the qregex patch just to add a little more flexibility to my ability to block mail at the level of my MTA. The latest version offers case insensitive pattern matching, logging of all matches, and a badhelo control file for checking the helo host of smtp conversations.

These patches can be found in the Arda Network Store. In addition, I’ve prepared a patch that combines the two together if you decide you want to use both.

Control Files

Whereas most of the control files are owned by root:wheel with permissions set to 644, the dh1024.pem, dh512.pem, rsa512.pem, and servercert.pem control files are owned by qmaild:qmail with permissions set to 600. This is to preserve the security of the Diffie-Hellman parameters and private keys used by qmail.

The Vmail System Account

Here you can see the .qmail files for the vmail account. For a further explanation of the .qmail files see the next section. The .tmda directory and the getvuserhome.sh and tmda.certkey.pem files are explained in the section dealing with TMDA. When using virtual domains, it is common practice to place all the mail accounts for a particular domain within a directory named after the domain. This is what I’ve done here. I have two virtual domains listed here, arda-homelinux-net and arda-homeunix-net, which correspond to the domains in qmail’s control files listed in the previous section. Here is a sample of what one of these directories looks like.

drwx------   8 vmail  vmail   512 Feb 13 23:06 andrew
drwx------  10 vmail  vmail   512 Feb 13 23:06 johanne

Each mail account has its own subdirectory within the domain directory. Giving each mail account its own subdirectory was convenient because it gave me somewhere to put account specific configuration files. It also helps to keep the mail accounts organized.

.qmail Files

The first file in this case is used to forward mail to a particular Maildir directory; I have it delivering mail to my account, andrew. This .qmail file is not likely to receive any mail directly as that would mean the mail was addressed to vmail which shouldn’t happen. The second file is what makes the first file work. The second file catches all mail addressed to an account not supplied with its own .qmail files. Therefore, mail addressed to andrew@arda.homeunix.net would be caught by the .qmail file in andrew’s mail account subdirectory while mail addressed to bogususer@arda.homeunix.net would be caught by the .qmail-default file in vmail’s home directory. It would then be handled according to the rules in vmail’s .qmail file. How qmail finds the proper mail account subdirectory of legitimate mail users is explained in the User Database section of this document.

The above explanation seems to imply that I don’t, in fact, need vmail’s .qmail file at all and could make due with just the .qmail-default file. That is probably true. However, all of the mail accounts use the two .qmail files and so I set up vmail the same way for consistency’s sake.

You’ll notice that vmail uses TMDA to filter incoming mail. This is what allows me to reject mail addressed to non-existant users on my domains but still receive mail to specific system aliases that don’t have .qmail files of their own. A more thorough explanation of this can be found in the sections dealing with aliases and the user database as well as the section on TMDA.

System Aliases

Because I’m using virtual domains with my system aliases, the recipient of all incoming mail is rewritten to the form <domain-name>-<recipient>. For example, mail received for postmaster@arda.homeunix.net will actually be delivered by qmail to arda-homeunix-net-postmaster. This confuses the alias directory no end and means that aliases for postmaster, or even arda-homeunix-net-postmaster, don’t work. This is the reason I had for setting up the .qmail and .qmail-default files in vmail’s home directory described in the previous section.

The User Database

The first two lines of the file don’t refer to regular mail users. The first line directs mail addressed to my system account (the account I use to log into my various machines) to my virtual mail account. The second line tells qmail where to find .qmail files for mail retrieved from my mail account on my ISP’s mail server. See the section Retrieving Mail from my ISP for more details. Notice that both these lines begin with a '='. This means that both require exact matches, no address extentions are allowed.

The last entry in the assign file is special. It is a catch-all line that directs mail not caught by a more specific entry to vmail’s home directory. Remember that this is where mail addressed to the system aliases will go and where I handle mail addressed to non-existant users with the help of TMDA. Notice that the last line of this file is a period. Don’t forget this period or the user database won’t work.

Some of you may have realized that I could have put all my system aliases into the user database and eliminated the need for TMDA to be involved in delivering mail to these addresses. For example, I could put the line:

=arda-homeunix-net-postmaster:vmail:398:398:/home/vmail/arda-homeunix-net/andrew:::

I used to have qmail set up so that all .qmail files resided in vmail’s home directory. Setting up the user database as described has allowed me to move most .qmail files out of vmail’s home directory and into the subdirectories of the mail accounts they pertain to. This has gone a long way to making my mail system more manageable as having all those .qmail files in one directory was making vmail’s home very crowded.

Run Scripts

It’s pretty simple. It sets up the needed PATH environment variable and sets the default delivery location for mail intended for local mail accounts. The defaultdelivery control file is not standard to qmail but its use is recommended by Life with Qmail. The default delivery location is not normally used as the .qmail files in each user’s subdirectory will be consulted by qmail to determine delivery instructions. If I forget to put a .qmail file in someone’s directory or if the file gets deleted, however, the default location acts as a safety net for mail delivery.

My qmail-smtpd run script is rather more complicated. It looks like this:

#!/bin/sh

QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`
MAXSMTPD=`head -1 /var/qmail/control/concurrencyincoming`
LOCAL=`head -1 /var/qmail/control/me`

if [ -z "$QMAILDUID" -o -z "$NOFILESGID" -o -z "$MAXSMTPD" -o -z "$LOCAL" ]; then
    echo QMAILDUID, NOFILESGID, MAXSMTPD, or LOCAL is unset in
    echo /var/qmail/supervise/qmail-smtpd/run
    exit 1
fi

if [ ! -f /var/qmail/control/rcpthosts ]; then
    echo "No /var/qmail/control/rcpthosts!"
    echo "Refusing to start SMTP listener because it'll create an open relay"
    exit 1
fi

exec /usr/local/bin/softlimit -m 4000000 \
/usr/local/bin/tcpserver -v -R -l "$LOCAL" -x /usr/local/etc/tcprules.d/tcp.smtp.cdb \
-c "$MAXSMTPD" -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp /usr/local/bin/rblsmtpd -b -r sbl-xbl.spamhaus.org \
/var/qmail/bin/qmail-smtpd arda.homeunix.net /usr/local/sbin/courierpasswd -s imap -- /usr/bin/true 2>&1

Like the qmail-send run script above, I modeled this run script after the one found in Life with Qmail. Like defaultdelivery, concurrencyincoming is not a standard qmail control file but is something added by Life with Qmail. Softlimit is part of the daemontools package while tcpserver and rblsmtpd are from the ucspi-tcp package. I’m using softlimit to cap the memory usage of qmail-smtpd. Tcpserver acts as a super-server and is intended to replace other super-servers such as inetd and xinetd.

Two things are of particular note with this run script. The -x option directs tcpserver to filter incoming SMTP connections through a database; tcp.smtp.cdb in this case. On my system, this database file is created by the second anti-spam program that I use, spfilter. The section on spfilter details how this file is created and what it does. It is rblsmptd that uses the information in tcp.smtp.cdb to actually make the decision whether or not to accept an SMTP connection.

The second thing to note is the program being invoked by qmail-smtpd, courierpasswd. This program authenticates users who try to send mail through my server to a remote domain. This is how I prevent my mail server from being used as an open relay. Courierpasswd allows me to access user account information using Courier authentication modules. Very convenient given that I use the Courier IMAP server. One consequence of this configuration is that no one can relay mail through my mail server unless they use a mail client that supports SMTP authentication.

qmail as a Null Client

Things I didn’t have to worry about on Europa were aliases, a user database, dot files, or virtual mail users. None of these items are necessary to run qmail as a null client. This really only left control files and a startup script to deal with.

Control Files

You’ll notice that there are fewer control files on Europa as compared to Callisto. Of particular note is the absence of a virtualdomains file. Also, the smtproutes file points to mail.arda.homeunix.net instead of my ISP’s SMTP server as it does on Callisto. Mail.arda.homeunix.net is an alias I’ve set up in my DNS server to point to Callisto. Some time ago I had occasion to change which of my machines was acting as the mail server and using an alias made the switch less complicated.

Also missing are the dh1024.pem, dh512.pem, and rsa512.pem control files. I can justify going to the trouble of maintaining these files on Callisto because of the volume of mail it handles. However, Europa’s volume is so low that it’s just not worth the effort.

As with Callisto, the servercert.pem control file is owned by qmaild:qmail with permissions set to 600. All the other control files are owned by root:root with permissions set to 644.

Run Scripts

This script starts up the processes necessary to send mail to Callisto. It also starts up processes used to deliver mail locally. These processes aren’t necessary as no mail is delivered locally but they don’t harm anything for being there. As with Callisto, I use programs from the daemontools package to control the qmail-send run script. I’m also using the defaultdelivery control file as I do on Callisto.

The IMAP Server

I wished to use virtual accounts for all the mail users on Callisto and Courier provides a number of ways to do this. I opted to use the userdb option since there is very good documentation on the Courier IMAP web site on how to do this. You probably already know that the IMAP server portion of Courier is available as a package separate from the full mail server package. I use the IMAP server package and this should be kept in mind while reading the following description. I will often refer to my setup as ‘Courier’ rather than ‘Courier IMAP’ simply to save keystrokes. Where I make reference to the full Courier package, I will make this clear.

Configuration Options

Take note of the --with-authchangepwdir option. This option is necessary if you want to make use of the password changing functionality of Courier authentication modules. This functionality is installed by default in the full Courier package but not in the IMAP package. Even with this option set, I still needed to manually copy the authdaemon.passwd program to the libexec/courier-imap/authlib directory.

Mail Accounts

You will notice that the entries for andrew, dhahn107, and tmda use imappw to identify the user’s password. This provides a way for Courier to restrict what a particular password can be used for. Other possible values are 'poppw' and 'systempw'. It is because I’m using imappw here that I need to include the -s switch with courierpasswd and courierpassd.

Courier includes very good documentation describing how to populate the userdb file. In my case, the reason the --with-userdb option above points to a file instead of a directory is because I’m using a small number of virtual domains. When setting up mail accounts for many virtual domains, it is often a good idea to point the --with-userdb option to a directory to provide a more manageable organization of your mail accounts. See the Courier documentation for more details.

One account, dhahn107, bears considering in more detail. Because it is so often the case that the user id of an email account matches the first part of the user’s email address (the part before the '@' sign) it is easy to forget that the two don’t have to be the same. When using multiple virtual domains, having the two always the same would impose an undesirable limitation. If the two were always the same, it would mean that I could not have the two accounts david@arda.homeunix.net and david@arda.homelinux.net on my mail server; it would be ambiguous which 'david' account I was sending mail to or from. With the user id different from the email address, I can have a 'david' account in each of my hosted virtual domains. Note that I still cannot have two 'david' accounts within the same virtual domain.

In my example, I’ve created a user id for David that consists of his initials, an acronym derived from the virtual domain he is a part of, and an arbitrary number just to ensure that the user id will be unique.

So why then do I have the user id andrew in my userdb file instead of asjahn001 you ask? Because I’m the sysadmin, and I make the rules.

It may be useful to note that the mail account user id appears nowhere else in my server setup. The only other place the user id will be used is by the user when logging into my server with his or her mail client.

Configuration Files

The only thing I changed here was where Courier looks for my server certificate. It’s the same certificate that qmail uses to establish secure connections.

Starting Courier

I used to use both init scripts to start Courier so it would listen on both the imap and imaps ports. I did this because, while the mail clients I use can speak SSL, the webmail package I use cannot. I have since configured Stunnel to secure the connection between my webmail and Courier so I no longer need to have Courier listen on port 143. Now I start Courier with the imapd-ssl.rc script only. I’ve removed the symlink in /usr/local/etc/rc.d that points to imapd.rc. My Stunnel Setup Doc has more information on how I’m using Stunnel.

The Webmail Interface

There really isn’t much to say about my SquirrelMail setup. I installed the software on Io, told it to use Courier as the IMAP server, told it that my SMTP server requires authentication and it worked. You can’t argue with results like that. One aspect of my SquirrelMail configuration that is peculiar to my setup is that I have SquirrelMail pointing to localhost for both my IMAP and SMTP servers with the smtp port set to 8025. My configuration is set up this way because I’m using Stunnel to secure the connections between SquirrelMail and my IMAP server and TMDA. See the section Daemontools on Callisto and my Stunnel Setup Doc for more details. Setting SquirrelMail’s smtp port to 8025 (used by TMDA’s smtp proxy) has also meant that I need to set up certain TMDA files even for users who don’t use this anti-spam software on my system. See the TMDA section for more details.

Because SquirrelMail is written in php, I needed to ensure certain parameters were set properly in my php.ini file. The settings that are relevant to SquirrelMail follow.

file_uploads = On
register_globals = Off
session.auto_start = 1
session.save_path = /var/php-sessions
session.use_cookies = 1
upload_max_filesize = 10M

drwxr-x--- 2 www www 512 Nov 9 18:55 php-sessions

By the way, I’m using php version 4.3.8 built as an Apache module.

As I mentioned previously, one of the nice things about SquirrelMail is the ability to extend its capabilities with the numerous plugins available for it. Here is the list of plugins I have working with my installation.

Name	Version
abook_take	core
calendar	core
change_pass	2.7
compatibility	1.3
compose_chars	0.1
folder_sizes	1.4
message_details	core
msg_flags	1.4.3
password_forget	2.0
quicksave	2.3
select_range	2.8.2
sql_squirrel_logger	1.1.0
squirrelspell	core
timeout_user	1.1.1
view_as_html	3.5a
vkeyboard	0.6
weather	4.1.1

The plugins with version numbers of 'core' are included in the SquirrelMail distribution.

One of the plugins listed in the above table is change_pass. As you may have guessed, this plugin provides mail users the ability to change their account password. Given that all mail accounts on Callisto are virtual, providing a means for users to change their password seemed like a good idea to me.

The change_pass plugin uses one of two programs (poppassd or courierpassd) to change the password of users. Both use the same network protocol to talk to clients making the two programs interchangable. Which program you choose to use depends on how you want to authenticate mail users. Poppassd uses PAM to change passwords. Since I use Courier as my IMAP server, I’m using the courierpassd program which uses configured Courier authentication modules to change passwords. Its use allows SquirrelMail to access Courier’s user database even though Courier is installed on a different machine from my web server. Another reason I’m using courierpassd is because I wrote it.

The big downside of using courierpassd in this way is that the connection between server and client is insecure, plain text user IDs and passwords are flying around the network when using this program. I’ve overcome this drawback by using Stunnel to secure the connection between SquirrelMail and my mail server.

Some of you familiar with Squirrelmail may be aware that Squirrelmail provides the option to use TLS secured connections between it and the IMAP and SMTP servers it talks to. All I have to do is enable these options and I don’t need Stunnel at all. All I can say is that when I started using Squirrelmail, it didn’t offer TLS secured connections. If I were setting up my servers now, I would likely use Squirrelmail’s native TLS support. Since I’ve already gone to all the trouble to set up Stunnel, however, I won’t be changing it until I need to.

Retrieving Mail from my ISP

Once Fetchmail was set up, I needed to create an entry in qmail’s user database and set up an account directory in /home/vmail/arda-homeunix-net just like I would with any other mail user account.

And that’s it. For very little effort, I am able to pull all of my mail delivered to my ISP’s mail server into my own mail server where I can manipulate it to my heart's content.

Starting Processes with Daemontools

Daemontools is a package that includes programs used to control the startup and shutdown of long-running processes. This package is intended to ensure that processes that are supposed to run all the time actually do.

Daemontools on Callisto

The qmail-send/run and qmail-smtpd/run scripts simply invoke the run scripts I descibed in the qmail run script section. The run scripts located in the log directories send all qmail logs to the directories indicated; in other words, qmail logs don’t show up in syslog. The setuidgid and multilog programs are also part of the daemontools package by the way.

Before I could start using multilog, I needed to create the directories it would be sending logs to. Here they are.

drwxr-x---  3 qmaill  wheel    512 Dec  9 10:19 /var/log/qmail
drwxr-x---  2 qmaill  wheel    512 Dec 13 15:03 /var/log/qmail/smtpd

The second supervise directory is located in /usr/local/etc and looks like this.

drwxr-xr-x  3 root  wheel  512 Dec 7 20:55 supervise
drwxr-xr-x  4 root  wheel  512 Dec 7 23:37 supervise/courierpassd
drwxr-xr-x  3 root  wheel  512 Dec 7 23:37 supervise/courierpassd/log
-rwxr-x---  1 root  wheel  181 Dec 8 00:02 supervise/courierpassd/run
-rwxr-x---  1 root  wheel   64 Dec 8 00:01 supervise/courierpassd/log/run
drwxr-xr-x  3 root  wheel  512 Dec 8 13:31 supervise/tofmipd
-rwxr-x---  1 root  wheel  152 Dec 8 13:41 supervise/tofmipd/run

And here are the run scripts.

File	Contents
courierpassd/run	#!/bin/sh exec /usr/local/bin/tcpserver -v -R -l callisto.arda.homeunix.net \ 127.0.0.1 pop3pw /usr/local/sbin/courierpassd -s imap --stderr 2>&1
courierpassd/log/run	#!/bin/sh exec /usr/local/bin/multilog t /var/log/courierpassd
tofmipd/run	#!/bin/sh exec su - vmail -c 'exec /usr/local/bin/tmda-ofmipd -f -p localhost:8025 \ -A "/usr/local/sbin/courierpasswd -s imap -- /usr/bin/true" \ -S /home/vmail/getvuserhome.sh' 2>&1

And here is courierpassd’s log directory as specified in the courierpassd/log/run script.

drwxr-x---  2 root    wheel      512 Dec  8 00:01 /var/log/courierpassd

There is no log directory under tofmipd because logging in tmda-ofmipd is really only designed to be used for debugging problems with the program.

Courierpassd’s run script is set up much like qmail’s smtprc script. It uses tcpserver to provide the network connectivity. Three things are of note in this file. One is that I specify the loopback address for courierpassd to listen on. But this doesn’t seem to make any sense because courierpassd must accept connections from Io, my web server, to do any good. I have courierpassd listening on localhost because I use Stunnel to create a secure connection between Callisto and Io. See my Stunnel Setup Doc for more details. The second point is that pop3pw is the alias given to port 106 in my /etc/services file. I’ve also seen poppassd used for this port. Make sure what you put here matches whatever is in your /etc/services file or just put the port number you want here instead. The third point is the --stderr switch used with courierpassd. This switch causes courierpassd logs to go to the directory specified in the log/run file instead of to syslog. This way, logs from tcpserver and courierpassd will go to the same place.

Before setting up Stunnel, I used tcpserver’s -x switch to restrict which IPs could connect to courierpassd. Now I’m using a client certificate with Stunnel which, at least in theory, is a lot more powerful than IP filtering.

I originally had courierpassd running under xinetd instead of supervise and tcpserver. However, I don’t use xinetd for anything else and I was already using supervise and tcpserver with qmail so I decided to do a bit of consolidation. There is an example of how to run courierpassd with xinetd in the documentation that comes with courierpassd.

Like courierpassd, tmda-ofmipd is listening on localhost as specified by the -p switch. It’s doing this for the same reason courierpassd is. My Stunnel Setup Doc talks about tmda-ofmipd too. Port 8025 is the default port tmda-ofmipd listens on and I saw no good reason to change that.

You will notice that I’m using two exec commands in the tofmipd run script as well as the -f switch to force tmda-ofmipd to run in the foreground. This is necessary to make tmda-ofmipd run properly as a supervised process.

Once all the supervise directories and run scripts were in place, I needed a way to start everything at system boot. To do this, I use svscan from the daemontools package. I created the directory /var/service and put symlinks into it that point to the various directories under my supervise directories. If that sentence confuses you, don’t worry, it confuses me too. So in the interest of clarity, here is what the /var/service directory looks like.

lrwxr-xr-x  1 root  wheel  38 Dec 7 23:36 courierpassd -> /usr/local/etc/supervise/courierpassd/
lrwxr-xr-x  1 root  wheel  32 Dec 4 00:00 qmail-send -> /var/qmail/supervise/qmail-send/
lrwxr-xr-x  1 root  wheel  32 Dec 4 00:02 qmail-smtpd -> /var/qmail/supervise/qmail-smtpd/
lrwxr-xr-x  1 root  wheel  33 Dec 4 13:31 tofmipd -> /usr/local/etc/supervise/tofmipd/

/usr/local/etc/rc.d/svscan.sh

#!/bin/sh

if ! PREFIX=$(expr $0 : "\(/.*\)/etc/rc\.d/$(basename $0)\$"); then
    echo "$0: Cannot determine the PREFIX" >&2
    exit 1
fi

# It would really, really be a Good Thing(tm)
# for you to enable some of the below
# control variables and the apropriate ulimit.
# These are only examples.
# Furthermore, you should think about additional
# limits you might need.
# Or, check login.conf(5) for a suitable
# alternative
#
# I really do suggest you use /var/service/ as your
# service spool directory. Check hier(7) for
# reasons.

# 10 Mb
#MINSEGMENT=10240
# 20 Mb
#MAXSEGMENT=20480
# 10 Mb
#MAXFILESIZE=10240
# 100
#MAXFD=100
# 40
MAXCHILD=40

# ulimits
#ulimit -d ${MINSEGMENT}
#ulimit -f ${MAXFILESIZE}
#ulimit -m ${MAXSEGMENT}
#ulimit -n ${MAXFD}
#ulimit -s ${MINSEGMENT}
ulimit -u ${MAXCHILD}

KILLALL=/usr/bin/killall
SVC=${PREFIX}/bin/svc
SVPROC=svscan
SVSCAN=$PREFIX/bin/svscan
READPROCTITLE=$PREFIX/bin/readproctitle
SVDIR=/var/service/

case "$1" in
  start)
        echo -n " svscan"
        exec env PATH=$PREFIX/sbin:$PREFIX/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH \
/bin/csh -cf "$SVSCAN $SVDIR |& $READPROCTITLE service errors: \
 ........................................................................................................... \
 ........................................................................................................... \
 ........................................................................................................... \
 ............................................................................... &" > /dev/null
        ;;

  stop)
        echo -n " svscan: "
        $KILLALL $SVPROC
        $SVC -dx $SVDIR/* $SVDIR/*/log
        ;;

  *)
        echo "Usage: $0 start | stop"
        ;;
esac

This script directs svscan to start supervise processes for all the run scripts it finds under the /var/service directory. A number of ulimit parameters are provided but I’ve only uncommented the MAXCHILD parameter in order to prevent fork bombs. You may want to explore the other ulimit parameters further to see if any are worth using. Using the script to stop svscan not only stops svscan itself but also every process with a run script under /var/service. Because of the way svscan.sh determines the PREFIX variable, you need to specify the full path when running this script manually.

Daemontools on Europa

The reason I have a /service directory on Europa and not a /var/service directory like on Callisto is because that’s where daemontools put it. On Callisto, I installed daemontools from the FreeBSD ports tree and the port maintainer has evidently set up daemontools to install the service directory under /var. I installed daemontools from a tarball on Europa and so service went into the root directory. I haven’t had any trouble with either setup.

Stopping Spam

Even though I don’t use this tcprules command any more, I thought someone might find it useful if he or she wishes to use one of the more aggressive blacklist source aliases but remove a small number of IP address ranges from the resulting block list.

TMDA (Tagged Message Delivery Agent)

Maildir is, obviously, where the mail goes. You’ll also notice the .qmail files which qmail is using to tell it how to deliver mail to this particular mail account. The contents of the .qmail files are explained in the relevant qmail section. All directories and files related to TMDA are in the .tmda directory. Below are listed the parameters I add to the default config file of each mail account that uses TMDA. I didn’t alter any of the lines already present in the default config file. In this example, the user Johanne Doe has the account user id of johanne and uses the email address johanne@arda.homeunix.net.

DATADIR = "/home/vmail/arda-homeunix-net/johanne/.tmda/"
FULLNAME = "Johanne Doe"
USERNAME = "johanne"
ADDED_HEADERS_CLIENT = {"X-Primary-Address" : "johanne@arda.homeunix.net"}
CONFIRM_APPEND = "/home/vmail/arda-homeunix-net/johanne/.tmda/lists/whitelist-append"
PENDING_WHITELIST_APPEND = "/home/vmail/arda-homeunix-net/johanne/.tmda/lists/whitelist-append"

LOGFILE_DEBUG = "/home/vmail/arda-homeunix-net/johanne/.tmda/log/debug.log"
#LOGFILE_INCOMING = "/home/vmail/arda-homeunix-net/johanne/.tmda/log/incoming.log"
#LOGFILE_OUTGOING = "/home/vmail/arda-homeunix-net/johanne/.tmda/log/outgoing.log"

The TMDA web site has great documentation on what all these parameters do so I’ll not repeat all that here.

Not all accounts on my system need to use TMDA. Because I have my webmail package pointing to TMDA’s smtp proxy, tmda-ofmipd, even these users need a minimal TMDA setup to be able to send mail from my webmail interface.

When I say minimal, that’s exactly what I mean. This is what the .tmda directory of an account not using TMDA looks like.

-rw-------   1 vmail  vmail  1972 Feb 13 23:13 .tmda/config
-r--------   1 vmail  vmail    41 Feb 13 23:06 .tmda/crypt_key

The crypt_key file is created that same way as for accounts that use TMDA. The config file has a single parameter in it.

ACTION_OUTGOING = "bare"

This parameter tells tmda-ofmipd to not add any tags to outgoing mail, which is what you want for an account that doesn’t use TMDA.

The only other difference compared to an account that uses TMDA is, of course, that the user’s .qmail files won’t include any mention of TMDA in them.

In addition to the virtual accounts, the vmail system account also has its own set of TMDA files and directories as well as .qmail files to control mail delivery. The two files, .qmail and .qmail-default, that vmail uses are explained in the .qmail Files section. This is what vmail’s .tmda directory looks like.

-rw-------    1 vmail    vmail        1955 Sep 13 08:00 config
-rw-------    1 vmail    vmail          41 Sep  8 08:15 crypt_key
drwxr-x--x    2 vmail    vmail        4096 Sep 27 15:16 filters/
drwxr-x--x    2 vmail    vmail        4096 Sep  8 08:09 lists/
drwxr-x--x    2 vmail    vmail        4096 Sep 30 01:05 log/
drwx------    2 vmail    vmail        4096 Sep  8 08:33 pending/
drwx------    2 vmail    vmail        4096 Oct 14 01:33 responses/
drwxr-x--x    2 vmail    vmail        4096 Oct 14 10:49 templates/
-rw-------    1 vmail    vmail           0 Sep 27 15:14 tofmipd

You will notice that this directory looks a lot like the /home/vmail/arda-homeunix-net/<username> directory described above except for the addition of the tofmipd file and the templates directory. The file is for storing user IDs and passwords of accounts that use tmda-ofmipd. I authenticate users with one of tmda-ofmipd’s other methods but the program complains if this file isn’t here. The templates directory allows me to use customized messages for automatic responses generated by TMDA which override the default messages.

The config file used by vmail is quite a bit different from that of a virtual mail account. Here is what it looks like.

TEMPLATE_DIR = "/home/vmail/.tmda/templates/"

#LOGFILE_DEBUG = "/home/vmail/.tmda/log/debug.log"
#LOGFILE_INCOMING = "/home/vmail/.tmda/log/incoming.log"

Most of the parameters found in a mail account’s config file aren’t here because mail won’t be sent from the vmail account directly. Another difference is that vmail doesn’t have its own Maildir; it never receives mail for itself. Vmail always fowards mail to another account or bounces it in the case of mail addressed to an unkown account.

The TEMPLATE_DIR parameter is set because I’m using a custom bounce template.

The really perceptive among you might be asking yourselves, ‘If the vmail account catches all mail not destined to a virtual account listed in qmail’s user database, doesn’t it foward mail addressed to every bogus account sent it’s way?’ The answer is; it would if not for my incoming filter file found in .tmda/filters. Here it is.

#       ~/.tmda/filters/incoming        filter incoming email

# bounces to bitbucket are dropped
to bitbucket@arda.homeunix.net drop

# accept bounced messages
from <> ok

# accept messages to root, postmaster, webmaster, and mailer-daemon
to root@* ok
to postmaster@* ok
to webmaster@* ok
to mailer-daemon@* ok

# bounce messages to all other mail accounts.
# entries in the qmail users database should
# catch mail to valid user accounts before this
# filter file is invoked.
to *@=arda.homeunix.net bounce
to *@=arda.homelinux.net bounce

I have only two virtual domains listed here but any number could be supported. Just add another 'to' line for each additional virtual domain.

We also need to look at TMDA’s global configuration file /etc/tmdarc. Here it is.

# tmdarc        TMDA global configuration file

SMTPHOST = "mail.arda.homeunix.net"
BOUNCE_ENV_SENDER = "bitbucket@arda.homeunix.net"

SMTPSSL = 1
SMTPSSL_CERTFILE = "/home/vmail/tmda.certkey.pem"
SMTPSSL_KEYFILE = "/home/vmail/tmda.certkey.pem"
PURGED_HEADERS = ["bcc", "resent-bcc", "user-agent", "x-delivery-agent", "x-mailer"]
SMTPAUTH_USERNAME = "tmda"
SMTPAUTH_PASSWORD = "<password>"

The SMTPSSL_* parameters are set here because I built qmail to use the starttls protocol and so TMDA can take advantage of a secure connection with my MTA. The certificate and private key are in a single file owned by vmail:vmail with permissions set to 400. The PURGED_HEADERS parameter removes the specified headers from a mail message before it is sent to the MTA. What the SMTPHOST parameter is doing should be self-evident.

The BOUNCE_ENV_SENDER parameter exists because there are many MTAs out there configured to reject mail with an empty 'Return-Path' header, which is what TMDA does by default when sending confirmation requests. Setting this parameter causes the Return-Path header to be filled with the specified value. It also provides a convenient way to identify bounces to messages generated by TMDA. For example, if a TMDA confirmation request bounces (which it will in the vast majority of cases when spam is involved) I don’t really want to see the bounce. That is why I drop mail that is bounced back to bitbucket in vmail’s incoming filter file detailed above.

The two SMTPAUTH_* parameters are necessary because, as explained in the MTA section, my qmail-smtpd requires authentication in order to relay mail through it. The two SMTPAUTH_* parameters are used by TMDA when sending confirmation requests and when dealing with undeliverable mail.

Note that the user 'tmda' has an entry in the userdb file described in the IMAP Server section.

Because the /etc/tmdarc file contains a plain text password enabling anyone to relay mail through my server, I’ve done my best to restrict access to the file. A listing of /etc/tmdarc looks like this.

-rw-r-----  1 root  vmail  388 Feb 17 06:41 tmdarc

The only reason I can get away with restricting read access to the file like this is because I’m using virtual mail accounts controlled by the single system account vmail.

The last thing to deal with is outgoing mail tagged by tmda-ofmipd. Don’t worry, the tricky parts were already covered when I explained how I’m authenticating users with Courier.

Tmda-ofmipd is an SMTP proxy. Mail clients that speak SMTP connect to tmda-ofmipd which manipulates a mail message according to the rules in a mail user’s outgoing filter file. This usually involves tagging the From: or Reply-To: headers in some way. Tmda-ofmipd then forwards the mail on to a regular SMTP server (in my case, qmail-smtpd). Tmda-ofmipd needs to authenticate users trying to send mail through it in order to work properly. How tmda-ofmipd authenticates users is given by the command I use to invoke the program. The command is invoked by the svscan program during system startup as explained in the Daemontools on Callisto section. Here is the tofmipd run script once again.

#!/bin/sh

exec su - vmail -c 'exec /usr/local/bin/tmda-ofmipd -f -p localhost:8025 \
-A "/usr/local/sbin/courierpasswd -s imap -- /usr/bin/true" -S /home/vmail/getvuserhome.sh' 2>&1

I’ve chosen to run a single persistent instance of tmda-ofmipd instead of invoking a new one each time a user wishes to send mail. There are three things to note in the above command. Tmda-ofmipd has a -u switch which tells it to run as a specified user. The -u switch does not seem to work when using virtual mail users controlled by a single system account. This is why my run script invokes su to make tmda-ofmipd run as user vmail.

The second point to note is my use of the -A option to authenticate users with a checkpassword compatible program called courierpasswd. This program requires that Courier (either the full package or just the IMAP server) be installed and that the accounts you want to authenticate against be accessible to Courier’s authentication modules. This is just what I want since I use Courier as my IMAP server and all virtual accounts are set up so Courier can authenticate against them. This is the same program that I’m invoking from my qmail-smtpd run script. Note that prior to version 0.70 of TMDA, I needed to use courierpasswd’s --stdin switch to have it work with TMDA. Version 0.70 and better no longer need this switch.

The third point is the -S option. This is how I tell TMDA where a user’s home directory is (and thus where that user’s TMDA config file is). This option is necessary because I have virtual accounts split into different directories according to domain and because a single system account controls all virtual domains. The shell script invoked, getvuserhome.sh, calls another program, courieruserinfo, that does the actual work of retrieving the user’s home directory. All getvuserhome.sh does is some formatting of the output. Courieruserinfo is another of the programs I wrote to allow me to use TMDA with my particular setup of Courier IMAP and qmail. Here is a listing of the getvuserhome.sh script.

-rwxr--r--  1 vmail  vmail   134 May  4 20:59 /home/vmail/getvuserhome.sh

And this is what the script looks like.

#!/bin/sh

SED=/usr/bin/sed

username=$1

homedir=`/usr/local/sbin/courieruserinfo -d $username | ${SED} 's/^home=//'`

echo $homedir

The reasons why I’m using the -f and -p switches of tmda-ofmipd are explained in the part of the Daemontools on Callisto section dealing with tmda-ofmipd.

I find this setup particularly convenient because it means that tmda-ofmipd doesn’t care where the user account information is stored so long as Courier can get at it. I also have to maintain only one mail account database.

All this talk of authentication is important because, as you’ll recall if you read the section on my MTA setup, I built qmail with the starttls-2way-auth patch and authenticating users is how I restrict who can relay mail through my server.

Before I managed to get tmda-ofmipd running using the daemontools package, I had it set up to start from a regular init script. In that scenario, I found that tmda-ofmipd would not detach from its controlling terminal properly which would result in two persistant processes instead of one; one running as vmail like it should, and the second running as root that shouldn’t have been there. To overcome this problem, I did a bit of hacking on tmda-ofmipd and added the -D option (for Daemonize). Here is the startup command that I used in my init script.

su - vmail -c "/usr/local/bin/tmda-ofmipd -D -A '/usr/local/sbin/courierpasswd -s imap --stdin -- /usr/bin/true' -c /home/vmail/arda-homeunix-net"

Although the patch I created for tmda-ofmipd is still available from my home page, I haven’t used it for some time. It may not be necessary with recent versions of tmda-ofmipd. Make sure you need it before using it.

And one more thing. Don’t forget to set the port of your SMTP server on you mail client to 8025 which is the port that tmda-ofmipd listens to by default.

Mail & DNS

How MX records are set up on Europa and Io, my primary and backup DNS servers, is detailed in my DNS & DHCP Setup Doc.

Software Home Sites