An recent email inquiry I received:
> I saw you had posted a reply to my inquiry about large installs running ECM2.
This mail server is my baby, so if I gush a bit, please forgive me.
> At this point, we’ve totally outgrown EIMS (as you can understand),
> and ECM2 is defintely the front-runner as far as replacements go. I
> have looked a lot at AtMail, which is basically a commercial ECM2,
SquirrelMail is what we use now for customer webmail, but we are seriously considering using something different and @mail’s webmail system might do the trick. We have some pretty sophisticated customers and a better webmail system is definitely needed.
> but decided that I really think I’d rather have a firm foundation and
> be able to modify it myself, instead of relying yet again on a
> commercial developer’s whims.
I hear you. EIMS lasted us a very long time, but we had to bite the bullet and make a change. I built a whole bunch of applescripts and php import scripts to migrate accounts over and that data then fed into a program that did syncing of mail over from EIMS mailboxes to the new ECM based system that mostly kept “read/unread” flags on the email. THAT was a big deal.
It’s been about a year now running on our server and it’s totally kick butt. We see almost no spam now, and the manageability is orders of magnitude more than it was with EIMS.
Most emails we get to make changes for email accounts we simply reply back to the admin of that account with the admin pwd and a URL to the mailadmin site. It has saved us mountains of support time.
The great thing about exim is that I can actually PROGRAM each phase of the SMTP conversation and the delivery phase to however complicated or personalized for each of our domains/users it needs to be.
Our exim config is definitely one of the more complex I’ve seen on the net.
And exim runs it without any issue.
> That said, it sounds like your ECM2 installation is handling your
> traffic well. May I ask what architecture you’re running your server
> on, and what types of loads you see?
OK, you asked…..
We run it on a Dual 1.0 Ghz Xserve G4 (10.4.8) with 2GB of ram, 2x60GB for boot and 2 x300 for data. Here is the Daily Load Average graph for that box:
Those spikes are when I’m running a mysqldump.
Here is the monthly graph:
we have:
419 sites
542 domains
1835 users defined
1069 of those defined as a mailbox.
Now, I took the package that George built and highly modified it. ECMs database schema and exim config is actually based on vexim, so there were some things I found on that site that I did pull into the system.
The primary difference on our system is that I abstracted domains out of the schema. I’ve always like the way EIMS implemented domain aliases. It so easily and transparently overlaid onto a “site”. So that’s what I did:
so, the primary unit is the site, then all the users and then you can have any number of domains on that site. Those graphics represent a state of the system over a year ago, there are quite a bit more fields in the sites table now, but this gives you the basic outline. the user can use any domain and can even use the % hack in their login id as well for all three services: SMTP, POP3 and IMAP. There is still a PRIMARY domain that you define in the site preferences.
> I have heard conjecture that
> Exim isn’t all that great under load,
We haven’t seen that and we run it on “old” hardware and it runs like a champ for us. I’ve implemented it for two other customers (one on Mac, one on Debian Linux) and we have two more lined up who want it.
The one thing I didn’t want exim to do was to handle outbound mail. The reason is that every message for delivery would have incurred another database lookup which would have caused unnecessary load and slow performance. So, we use an instance of Postfix with a very light config to handle all that delivery. This could be setup on the same box by having Postfix listen on a different port, but we already had in place an existing system on a different server, so we just used that.
> but I’ve also heard version 4
> took care of a lot of that. I don’t think we’re huge load here, but
> we do do at least a couple thousand messages per hour.
ok, I looked at our graphs of connections:
and taking the hour of 14:00 Dallas time, which seemed the busiest…..
received mail: 587
received mail: 1172
Now, I was able to pull those numbers very quickly because we LOG all blocks and all accepted messages into mysql:
SELECT count(*) FROM `recent_mail` WHERE recvd BETWEEN ‘2007-03-19 19:00:00’ AND ‘2007-03-19 19:59:59’
SELECT count(*) FROM `block_log` WHERE `when_blocked` BETWEEN ‘2007-03-19 19:00:00’ AND ‘2007-03-19 19:59:59’
(We store ALL times in UTC and then adjust for the user when displaying through the web admin pages)
The admin of an site or even an individual account holder can now see with their own eyes what emails were blocked and why.
The block log:
The Recent mail log:
I also built into it:
- greylisting
- greylisting exceptions by:
- site (stable through whatever domains are on that site)
- source IP or IP range
- sender domain or sender email
- recipient domain or recipient email
- auto blacklist of ips:
When an incoming server tries to helo with one of my ips or one of my names, that IP address is automatically added to the blacklisted hosts table with an expiration for a month.
- spam assassin:
- sql based bayesing scoring
- sql based auto white listing (the more mail you get from a sender, the lower their email is rated for spam)
- global, site and user level based prefs
- connection logging/profiling
I keep track all every single IP address that connects to our server.
As they progress through a SMTP connection, I update certain values on that record:
ip cnxn_count, cnxn_first, cnxn_last, reverse_ok_count,
helo_ok_count, quit_count, bad_from_count, bad_rcpt_count,
ok_rcpt_count, dnsbl_block_count, last_dnsbl_time, last_dnsbl
- whitelisting (globally or per site):
- by sender
- by recipient
- by ip or ip range
I also implemented catchalls similar to the way that EIMS has them:
Through the use of a preferences table, I can also selectively turn on/off certain features of the mail server in real-time without touching the config:
allow_cnxns allow_trusted allow_authd allow_other greylist_on
Also I liked the options for users that EIMS had: mailbox, forward and both, but ours has expanded features:
we also now have TLS based SMTP, POP3 and IMAP using a self-signed certificate.
> Thanks for any input you may be able to provide!
Here is a screen shot of the admin interface. ( I completely rebuilt the ECM web admin interface to handle more features and deal with the changes in architecture)
The Blocks column is the number of blocks in the past 24 hours and the last hour.
The Recent column is the number of accepted msgs in the past 24 hours and the last hour.I also wrote a bunch of support scripts that tail through some of the logs and update the login times for those users in the database.
Things I haven’t implemented but are mostly already built, just need testing:
- logging of email sent by authenticated users.
- automatically feeding email sent to spam traps into Spam Assassin for bayesian scoring
- Archiving of email per site or per user for corporate entities required to do so.
So, there it is.
Hope that answers all your questions and more!