Sendmail is the most prevalent Unix MTA. It is the oldest MTA and is still one of the most powerful ones that are available. However it has never been known for being secure.
http://lwn.net/Articles/176596/
Most of it’s bad reputation comes from regularly having serious security holes. The above URL has the most recent one. Neither Qmail nor Postfix has had a serious security issue. Dan and Wietse appear to have aggressively audited each other’s code in an attempt to find such a hole without success.
Sendmail was initially designed with a single process running as root which does everything. Any bug in that program and you lose. In recent times you have two processes, one of which doesn’t run as root. This alleviates the problem but doesn’t compare to the 10+ programs that may be run for different tasks on a Postfix or Qmail system, of which only two will have root access (the local delivery process and the master controlling process).
Another part of the Sendmail problem is the crufty old code. Exim has a similar design to Sendmail in terms of process duties, but has a much better security history due to being written more recently.
On many occasions over the last ~8 years I have had debates with Sendmail advocates regarding the security issues. The Sendmail advocates have consistently claimed that all the bugs are fixed now and Sendmail is only attacked because it’s popular. Given the track record it seems that it’s a bad idea to claim that the security flaws have all been fixed.
In regard to the popularity issue we have to keep in mind that fact that Windows has a much larger user-base than Linux. Any argument that you might make in favor of Sendmail over Postfix in terms of security flaws being a function of popularity is an argument in favor of Windows over Linux. I find it particularly amusing when BSD users claim that Sendmail only gets cracked because it’s popular. What does that say about the security of BSD given that BSD is much less popular than Linux?
On many occasions people have pointed out to me that you can run Sendmail as non-root. Almost 10 years ago I wrote a web page describing how to do this. Doing that has always been a hack, although it should work reasonably well for a machine that only runs Sendmail as an outbound relay.
Sendmail was a nice MTA in the early 90’s. But it’s time has passed. Let’s all upgrade to mail server software that doesn’t require regular security updates. Sendmail and Exchange belong in a software museum, not on the net.