Replicating IMAPs (dovecot) mails folders and sharing (through ownCloud) contacts (kmail, roundcube, etc)

dual IMAPs servers:

Having your own server handling your mails is enabling -you can implement anti-spam policies harsh enough to be incredibly effective, place catch-alls temporary addresses, etc. It does not even require much maintainance these days, it just takes a little time to set it up.

One drawback, though, is the fact if your host is down, or simply its link, then you are virtually unreachable. So you want a backup server. The straightforward solution is to have a backup that will simply forward everything to the main server as soon as possible. But having a backup server that is a replica of the main server allows you to use one or the other indifferently, and definitely have always one up at hand.

In my case, I run exim along with dovecot.  So once exim setup is replicated,  it’s only a matter of making sure to have proper dovecot setup (in my case mail_location = maildir:~/.Maildir:LAYOUT=fs:INBOX=~/.Maildir/INBOX
and mail_privileged_group =   mail  set in /etc/dovecot/conf.d/10-mail.conf along with ssl = required in /etc/dovecot/conf.d/10-ssl.conf  – you obviously need to create a certificate for IMAPs, named as described in said 10-ssl.conf but that’s not the topic here, you can use only IMAP if you wish).

Then, for each user account (assuming we’re talking about a low number), it’s as simple as making sure SSH access with no passphrase can be achieved from one of the hosts to the other and adding a cronjob like:

# UNRELIABLE DOVECOT TOOL */2 * * * *     user   dsync -f mirror secondary.domain.net 2> /dev/null
*/2 * * * *     user   isync --all --create-remote --quiet 2>/dev/null

isync requires a ~/.isyncrc such as:

MailDir ~/.Maildir
Delete yes
Expunge yes
OneToOne yes
Tunnel "ssh -q secondary.domain.net /usr/lib/dovecot/imap"
RequireSSL no

The first run may be a bit slow but it goes very fast afterward (I do have a strict expire policy though, it probably helps). This isdone the the primitive  way, recent version of dovecot (ie: not yet in Debian stable) provides plugins to do it.

You may as well install unison on both server and synchronize things like ~/.procmailrc, /etc/aliases or whatever, for instance:

8 */2 * * *	user	unison -batch -auto -silent -log=false ~/.procmailrc ssh://secondary.domain.net//home/user/.procmailrc 2> /dev/null

Once you checked that you can properly login on both IMAPs, it’s just a matter of configuring your mail clients.

and many mail clients:

I use roundcube webmail whenever I have no access to a decent system with a proper mail client (kmail, gnus, etc) configured. With two IMAPs servers, there’s no benefit of not having the same webmail setup on both.

The only annoying thing is not to have common address book. It’s possible to replicate the roundcube database but it’s even better to have a cloud to share the address book with any client, not doing some rouncube-specific crap. So I went for the option of installing ownCloud on one of the hosts (so far I’ve not decided yet if there is a point in replicating also the cloud, seems a bit overkill to replicate data that is already some sort of backup or replica), pretty straight-forward since I already have nginx and php-fcgi running. And then if was just a matter of pluging roundcube in ownCloud through CardDav.

Once done, you may just want to also plug your ownCloud calendar/addressbook in KDE etc, so all your mail clients will share the same address book (yeah!). Completely unrelated, add mozilla_sync to your ownCloud is worth it too.

The only thing so far that miss is the replication of your own identities – I haven’t found anything clear about that but havent looked into it seriously. I guess it’s possible to put ~/.kde/share/config/emailidentities on the cloud or use it to extract identities vcard but I’m not sure a dirty hack is worth it. It’s a pity that identities are not part of the addressbook.

(The alternative I was contemplating before was to use kolab; I needed ownCloud for other matters so I went for this option but I keep kolab in mind nonetheless)

 

Update 1: Stop using dsync that is tremendously unreliable as of today, use isync instead.

Expiring old mails on (dovecot IMAPS) server side

Years ago, I was using gnus to read my mails: among other things, I liked the fact that it was, by default, as expected from a newsreader, only showing unread messages and properly expiring old messages after some time.  Then, using KDE, at some point, I switched to Kmail because of this nice integration within the desktop environment. Obviously I had to configure it to remove old mails (expires) in a similar fashion.

Then Kmail2 arrived. I’m not able to use this thing. It either does not even start or start overly slowly and use up 100% of cpu time for minutes, whatever computer I’m using, whether it’s an old bold regular P4 or an Athlon II X4, whether I have 1GB RAM or 8. I gather it’s related to akonadi/nepomuk/whatever, stuff supposed to improve your user experience, with fast search and so on. Fact is it’s unusable on any of my computers. So I end up, these days, using Roundcube webmail, which is not that bad but makes me wonder whether it’s worth waiting for Kmail2 to be fixed and, worse, leaves me with IMAPS folders with thousands of expired messages that should be removed.

So this led me to consider doing the expires on the server side instead of client side, with my user crontab on the server. Logged on the server, I just ran crontab -e and added the following:

# dovecot expires (SINCE means: received more recently than)
# not flagged and already read, 1 week old min
05 */5 * * *	/usr/bin/doveadm expunge -u 'user' mailbox '*' SEEN not FLAGGED not SINCE 1w
# not flagged nor read, 8 weeks old min
09 */5 * * *	/usr/bin/doveadm expunge -u 'user' mailbox '*' not SEEN not FLAGGED not SINCE 8w
# read junk, 2 hours old min
15 */5 * * * 	/usr/bin/doveadm expunge -u 'user' mailbox 'Trash/Junk' SEEN not SINCE 2h
# unread junk, 2 days old min
25 */5 * * *	/usr/bin/doveadm expunge -u 'user' mailbox 'Trash/Junk' not SEEN not SINCE 2d

(Obviously you want to replace user by your local user account and Trash/Junk by your relevant junk IMAP folder) . This setup could probably be enhanced by using flags like DRAFT and such – however, on my local server, no actual draft got properly flagged as such, so it’s better to rely on the basic mark FLAGGED.

Securing and improving internet services, including SSH and SMTP, using xinetd

As stated by its manpage, xinetd performs the same function as inetd: it starts programs that provide Internet services. Instead of having such servers started at system initialization time, and be dormant until a connection request arrives, xinetd is the only daemon process started and it listens on all service ports for the services listed in its configuration file. When a request comes in, xinetd starts the appropriate server. Because of the way it operates, xinetd (as well as inetd) is also referred to as a super-server.

The X in the name stands for extended. Which means the following is really for xinetd, not openbsd-inetd 🙂

Still according to its manpage, so far, the only reason for the existence of a super-server was to conserve system resources by avoiding to fork a lot of processes. While fulfilling this function, xinetd takes advantage of the idea of a super-server to provide features such as access control and logging. Some people will say, and they’ll be right, that running all services through a wrapper implies, instead of conserving resources, somekind of overhead: in conserves resources since it avoids running concurrently as many services as available on the server, right, but this makes no sense on a server with a wide audience that actually have enough users so all services are anyway almost always running concurrently. In this case, some people would be correct to assume more efficient to use standalone servers for each service .

But, and that’s the point, access control makes a difference. Sure, standalone servers have also access control. OpenSSH does ; and using OpenSSH via xinetd should not discourage to look into /etc/ssh/sshd_config. Nonetheless, xinetd access control applies to any service running through it. And that’s prett-ay, pretty good. For instance, you do not need to configure each standalone server to be hardened enough against DDoS, if xinetd is, you should be fine.

So let’s get to business. We assume here you have xinetd up an running. Shouldn’t be a big deal, xinetd is standard on many GNU/Linux systems.

Normally, you should have a /etc/xinetd.d where you can add bits of config for xinetd (if it does not exists, well, you could still use the default config file /etc/xinetd.conf).

There you have basic standard stuff: chargen, daytime, discard, echo, time. If you do not want to provide these, sure make sure each entry in these files got the line:

disable = yes

For each following example to work, you must indeed shut down the standalone server, otherwise the service port won’t be available to xinetd. Also, in the following examples, you’ll have to edit the IPs according to your network.

This is for OpenSSH, with a specific port for root login (probably a nuisance on a distant server supposed to be frequently accessed as root – but a safe pick for a local network server rarely accessed as root from the web):

# To work, sshd must not run by itself,
# so /etc/ssh/sshd_not_to_be_run
# should exists

# allows unrestricted SSH only to local network
service ssh
{
socket_type = stream
protocol = tcp
wait = no
user = root
bind = 192.168.0.1
only_from = 192.168.0.0
server = /usr/sbin/sshd
server_args = -i
}

# allows SSH from the web but restricted to users listed
# (root being forcefully disallowed)
# restrict also to only 5 connections per IP (per-source)
# and limit the rate of incoming connections (cps)
service ssh
{
socket_type = stream
protocol = tcp
port = 22
wait = no
user = root
bind = 88.???.???.???
server = /usr/sbin/sshd
server_args = -i -o PermitRootLogin=no -o AllowUsers=thisuser
cps = 30 10
per_source = 5
log_on_success = HOST USERID
}

# allow SSH from the web only for root, on port 33333
# requires /etc/services to include lines:
# rootexternalssh 33333/tcp
# rootexternalssh 33333/udp
service rootexternalssh
{
socket_type = stream
protocol = tcp
port = 33333
wait = no
user = root
server = /usr/sbin/sshd
server_args = -i -p 33333 -o AllowUsers=root
cps = 30 10
per_source = 3
log_on_success = HOST USERID
}

This is for Dovecot, an IMAPS server. This setup listen on the local network. You can easily tune it following the example given above:


service imaps
{
socket_type = stream
protocol = tcp
wait = no
user = root
bind = 192.168.0.1 127.0.0.1
only_from = 192.168.0.0 127.0.0.1
server = /usr/lib/dovecot/imap-login
flags = IPv4
server_args = --ssl
}

I won’t provide an exhaustive list of services that you can run with xinetd. You can surely find for yourself what suits you best! 🙂 But the presentation wouldn’t be complete if I missed traps. Yes, you can set up traps with xinetd. For instance if you do not use ftp, irc, telnetd, etc, you can safely assume that someone trying to connect on these services ports is trying to do something he shouldn’t. And you can then decide to disallow further connections.

# bind must be set so we do not shut off clients from the
# local network that made dumb scan
service ftp
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

service sftp
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

service irc
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

service telnet
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

These are basic examples. You can do more.

For instance, years ago, I wrote SeeYouLater, denying access to spammers with hosts.deny, a production-ready software that looks in the SMTP daemon logs for identified spam sources IPs and then ban them via /etc/hosts.deny (which xinetd handles).
With the SMTP daemon run through xinetd, any identified spam source will no longer even be able to connect (which presents plenty of advantages, as it is costless by comparison to discarding the spam sources during SMTP transaction).
SeeYouLater depends on perl and MySQL, the debian apt source is

deb http://dl.gna.org/seeyoulater/ ./

and there is a cookbook entry covering setup with Exim.

I’d like to mention that I ran xinetd on Gna! main servers (for mail with exim, or for CVS/SVN/Arch, etc – mail server was moved and I did not follow the way it is set up) and the overhead mentioned above was unnoticeable while the number of connections per minutes was quite higher than what you would expect on a small business network server or whatever.