|
|
A client is considering some options for serious deployment of some CPU intensive work. The options that are being considered include cloud computing (Amazon EC2 [1]), virtual machines (Slicehost [2] and Linode [3]), and purchasing servers to install in racks at various locations. I can’t disclose the criteria that will determine when each of those three options will be used (I expect that we will end up using all of them). But my research on the prices of various servers will hopefully be useful to someone.
For the server vendor I chose Dell. I believe that HP offers slightly better quality hardware than Dell, but they cost more and are more difficult to deal with (I can’t even get a price online). For this project I will be using a bunch of redundant servers (in a similar concept to the Google server array) so I’m not going to be overly bothered about losing a server occasionally – therefore the slight benefit that HP offers for reliability does not make up for the expense.
Dell has some 1RU servers that have two CPU sockets and allow eight CPU cores. It seems that the best value that Dell offers for a server without RAID (the entire server is redundant) is a PowerEdge SC1435 that has two Opteron 2352 quad-core CPUs running at 2.1GHz, 4G of RAM, a 1TB SATA disk, and a Broadcom PCIe Gig-e card for $3,816.50. That machine gives an option of 2.3GHz CPUs for an extra $621.50, I am not sure that increasing the clock speed by almost 10% for a 16% increase in system price is a good idea.
The second best option was a PowerEdge 1950 III that has two Xeon E5420 2.5GHz quad-core CPUs with 12M of cache, 4G of RAM and a 1TB SATA disk for $4,302.30. The Intel option has 3 years of support included while the AMD option included 1 year of support and needed at least an extra $990 for 3 years of support. So it seems that if 3 years of support is desired then the Intel based server becomes significantly cheaper and is probably a better option.
Dell’s 2RU and 4RU servers are of no interest if you want CPU performance. The 2RU servers only support two processors and the 4RU servers only support four processors. So it’s a ratio of 2 processors per RU for 1RU servers vs one processor per RU for 2RU and 4RU servers, and the 2RU and 4RU servers are a lot more expensive too.
I am investigating the Dell blade server. Blade servers are great for CPU density and good for management. The Dell blade enclosure M1000e takes 10RU of space and supports 16 half-height blades or 8 full-height blades. The Dell M905 blade supports four AMD quad-core processors for a total of 128 cores in 10RU, there are also half-height blades that support two quad-core processors for the same CPU density.
So in terms of CPU density it’s an average of 12.8 cores per RU for the blade server vs 8 cores per RU for 1RU servers. While I haven’t got a complete price yet, it seems that four CPUs suitable for the M905 will cost about as much as four 1RU servers. So the 1RU systems are definitely better value for money than the blade server. The difference is the management cost. N servers that have two CPUs will be more work than N/2 servers that have four CPUs, but on the other hand blade servers require some specialised skills to run them (which I don’t have) and that might also cause problems. I don’t think that blades will be part of this project.
When reading a magazine I often see an advert for a product that I want to buy (or recommend that a client buy). This is of course expected as the advertisers put a lot of effort into targeting their adverts to the people who read such magazines. However I often decide that I want to buy the product some weeks after reading the magazine.
Linux Journal [1] usually has adverts for good server hardware that supports Linux. Their web site is quite nice in many ways, it allows subscribers to read articles online and has an index to all back-issues. But it has no link that I could find for reading the adverts! I would like to see an index of all advertising that has ever been published so that I can read the adverts online. The ancient adverts are good for historical reference and the new ones are good for purchasing decisions.
Right now I would like to be suggesting that a client consider buying a number of servers from a company that advertises in Linux Journal, but I’m not because they don’t publish their adverts online!
To make matters worse the Linux Journal web site doesn’t include a suitable contact address for issues unrelated to subscriptions. I believe that a magazine needs to publish an email address for copyright infringement reports (I often notify magazines when someone rips off their content so that they can issue a DMCA take-down notice), an email address for technical problems (I often notice rendering errors and broken links in web pages and like to report them), and an email address for random stuff. I’m even happy to use a web-based form to submit my suggestions if that’s what makes them happy, but having no published way of notifying them is simply a mistake.
NB I do have some email addresses of LJ employees in my addressbook somewhere, I will send them email if no-one responds to this blog post. But I’m publishing this because it’s probably a mistake that other magazines make, and because any response from them is going to be too late for the report on hardware prices that I’m writing.
Update: Linux Journal does allow you to download a sample copy, that is a PDF of one complete issue that includes adverts. So I can see adverts from August 2008.
An article from 1999 suggested that car crashes caused a financial loss in OECD countries equivalent of 2% of their entire economies [1]. An article from the Sydney Morning Herald in 2001 gave a conservative estimate of the cost of a road fatality at $1.5 million [2], it also notes that due to different analysis methods American transport economists derived a figure of $5.5 million. $1.5 million in 2001 adjusted for CPI would be close to $2 million now.
Currently that $2M cost is an externality of the car industry. Most of it is paid by the government, IE we all pay for it through our taxes. This means that there is little financial incentive for drivers and car companies to make the roads safer. Many of the attempts to legislate road safety fail due to the legal system being unable to manage the rapidly changing range of vehicles on the market.
The insurance companies have very detailed analysis of the relative safety of vehicles, so it seems that the only sensible way of enforcing safe driving is through economic measures implemented via insurance.
I believe that for every person who is killed or seriously injured on the road a fine of $2M should be levied. Every driver should be compelled to have insurance to cover such fines (driving without insurance should be illegal).
Then the government could cease being involved in regulating what types of car someone can drive. If someone who is less than 25 years old can get insurance for a turbo-charged car then it probably means that a statistical analysis suggests that the combination of driver and vehicle is likely to be reasonably safe (EG there are many turbo-charged cars on the market that are not particularly fast).
Now this will increase the car insurance costs for everyone, but it will decrease the amount of general tax money that is spent on issues related to road fatalities, which would allow the income tax rates to be decreased. This means that any tax-payer who has a good driving record and who drives a type of car that tends not to be crashed could expect to save money overall. Any tax-payer who doesn’t drive a car would save even more money.
But the main point of this idea is to increase road safety by forcing bad cars and drivers off the road. Currently defective cars are only removed from the road if police notice something unsafe about them and cite them for being unroadworthy – this only happens if it’s a problem which can be observed from outside the vehicle (EG worn tires or broken lights). In some states elderly drivers have no requirement for periodic health checks to determine their ability to drive, I know of one case of a woman who was certified as legally blind, ordered a white cane, and then drove home afterwards! I’m sure that insurance companies would implement whatever tests are necessary to reduce the risk of being hit by multiple $2M fines from a single crash.
Bruce Schneier writes about the risks involving children abandoned in cars and cites an article about the tragic deaths of children in hot cars [1]. One unfortunate error that he made was to not cite the following from the end of the last page of the Washington post article he cited [2]:
In hyperthermia cases, he believes, the parents are demonized for much the same reasons. “We are vulnerable, but we don’t want to be reminded of that. We want to believe that the world is understandable and controllable and unthreatening, that if we follow the rules, we’ll be okay. So, when this kind of thing happens to other people, we need to put them in a different category from us. We don’t want to resemble them, and the fact that we might is too terrifying to deal with. So, they have to be monsters.”
I believe that similar thought processes are used in relation to many other situations, and that such thought processes prevent people from taking appropriate actions to minimise the risk. If someone considers that forgetting a child in the back seat to be an accident that could happen to anyone then they would be inclined to take action to minimise the risk (such as spending some money on a sensor). If however they consider such forgetfulness to be proof of being a “bad parent”, then as they are a “good parent” they would have to avoid buying a monitor. I’m surprised that Bruce didn’t draw an analogy between this and the forgetful losses of laptops and guns by people who work for law enforcement agencies (which he has written about before).
I wonder how expensive it would be to make a sensor for heart-rate, breathing, and temperature integrated with a GSM modem and a GPS? If it could be small enough to be attached to clothes then the child could wear it at all times.
If such a sensor was to detect a sign of a problem it wouldn’t matter whether the child was forgotten in a car, at day-care, or even being actively supervised. The data would be sent to the monitoring agency along with GPS data. The monitoring agency could then phone the parents. If the parents don’t answer or don’t know where the child is then the police could track down the GPS location. Probably most calls would be due to parents leaving a child too close to an air-conditioner or playing outside in the sun in summer which are unlikely to give a fatal result and a phone call would get a quick fix for what would only be a minor health problem.
If the device was marketed as monitoring for “sleep apnia” then parents could buy it without admitting to the possibility that they might do anything wrong. The causes of SIDS are a topic of ongoing research and parents can admit to being worried about their children suffering from it without admitting any possibility that they might make a mistake.
I am not aware of any Linux Users Group (LUG) being active in informing it’s members of how the policies of the various political parties compare with regard to free software and the other issues that are of interest to most members. I believe that this is a grave mistake.
Shortly before an election there are many social groups that send lists of questions to all the parties. They ask about the policies the parties have in regard to the issues that they care about, and helpfully mention the number of members that will receive the response. This of course doesn’t mean that every member of the group in question will cast their vote in the same way, merely that they will take note of the answers.
The committee members of the parties in question will then decide how to answer the questions and whether policy should be tweaked to allow answers that the lobby groups will like. So this process not only helps members of a group make informed voting decisions related to issues that they care about, but it also helps political parties choose policies that are least offensive to the group in question.
Here is a draft of a list of questions that I think should be asked of all political parties on behalf of Linux users:
- It is important for all citizens to access all government data without being forced to buy new software or hardware, open standards allow everyone to access the data with free software. Do you support the use of open standards for data on government web sites and other forms of electronic communication between government agencies and citizens?
- For long term archival of records it is important that file formats remain readable. The only effective way of doing this is to use open file formats that are implemented in free software. Do you support mandating that all data submitted to government agencies (by citizens or corporations) be in open file formats wherever possible?
- In these difficult economic times there is a great interest in keeping jobs in the country instead of sending money overseas. To what extent do you support the use of free software that is installed and managed by locals (keeping the money in the economy) instead of importing software at great taxpayer expense?
- Commercial software has a limited support period, after that time has elapsed there is no further support and systems become increasingly unreliable. Do you support mandating that all systems relating to the emergency services run on free software to allow quality long-term support by local citizens?
- There has been a lot of concern recently about the spread of child-porn. The best available evidence shows that insecure home PCs that run “Trojan Horse” programs are a key part of distributing it and other illegal material. Do you support the introduction of government programs to train parents in installing one of the more secure free operating systems on their home PC to protect their children?
This is just a rough draft. Obviously there needs to be local differences (EG don’t use point 3 in the US because MS brings money into the US economy).
Does anyone have any suggestions for other questions?
I have just been asked for advice about “secure filesystem” and decided to blog my answers.
The first issue is what is meant by “secure filesystem, that could either mean the ability to restrict file access (EG by supporting SE Linux security contexts and using SE Linux for file access control) or the ability to encrypt data in case the machine is stolen. For access control I recommend SE Linux of course. For encryption on a local machine I mostly use dm-crypt which is configured with the cryptsetup utility. I encrypt at the LVM logical volume level as it is common that there are some LVs that don’t need to be encrypted. For files that need extra encryption or files that are shared between machines I use GPG.
A question was asked about kernel vs user-space filesystem encryption. AES is in the kernel so there is no lack in terms of strong encryption there. Also performance is pretty good (in most cases the CPU is fast enough that the hard drive is the bottleneck). For fine grained encryption (such as some of the experimental filesystems that encrypt data separately for each user) user-space is probably the only way to go.
If you want servers to be “high-security level” and protected from “hackers or unauthorised people” then it’s difficult to offer any advice that is smaller than a text book. I suggest that if you have such questions then you should do one of two things. If you are running a corporate IT department then hire an expert who can help with determine your specific requirements and meet them. If you want to learn about computer security and run your own systems in the best way possible then read as much from the experts as possible.
If you are looking for a project to contribute to related to security then if you choose SE Linux I could offer some specific advice on things that need work. I suggest not deciding on whether to do “kernel level or user level” work up front, but decide first which area of security you want to work on and then select a project which fits – then you should be able to determine whether your skills are best suited to kernel or user space coding. As for whether developing a new filesystem is necessary, I will note that SE Linux works well on Ext3 and XFS, it has just become usable on JFFS2, and it will work on other newer filesystems in the near future. Adding SE Linux support to a filesystem is not a difficult task if the filesystem supports XATTRs. I believe that there is a lot of scope for other access control systems to be developed which use XATTRs for security labels.
I can’t advise on e-books. I generally don’t read books, I read blogs and papers. Anything that I read which I consider to be worth recommending will probably have a link from my blog.
I’m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2].
The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn’t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn’t. It often refuses to work as desired, gives no good explanation, and it’s data structures can’t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well – it’s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it’s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them.
Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv).
Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed – and it’s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named “new“, “cur“, or “tmp” which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well!
A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it’s because not many people use Cyrus that such things don’t get fixed.
One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can’t think of them at the moment.
Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.
Continue reading Why Cyrus Sucks
I have recently configured my mail server to use IMAP. I started doing this when I was attending Linux.conf.au so that I could read urgent mail using my EeePC while at the conference and then be able to deal with the more complex stuff using my laptop later on.
The next logical step is to have mail delivered to different folders in the IMAP account. While there are ways of doing this via the Subject and other header fields, my needs are not that great. All I need to do is to support user+extension@example.com going to a folder named extension in the user’s mail store. While changing my mail server I decided to install Postfixadmin at the same time.
My first attempt to use Maildrop was to put the following in the /etc/postfix/main.cf file:
mailbox_command = /usr/bin/maildrop -d mail -f “$SENDER” “$DOMAIN” “$USER” “$EXTENSION”
That seems to only work when you have local accounts, so I ended up setting fallback_transport = maildrop and then putting the following in /etc/postfix/master.cf:
maildrop unix – n n – – pipe flags=DRhu user=vmail argv=/usr/bin/maildrop -d vmail ${nexthop} ${user} ${extension}
Where vmail is a Unix account I created for storing mail. Then I added the following to /etc/postfix/main.cf. Some of these are probably redundant (such as the virtual_mailbox_base). The recipient limit is set to 1 because there are no command-line parameters for maildrop to support two recipients for the same message.
virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_mailbox_domains = mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_gid_maps = static:2000
virtual_uid_maps = static:2000
virtual_mailbox_base = /mail
vmaildir_destination_recipient_limit = 1
virtual_transport = maildrop
maildrop_destination_recipient_limit = 1
The files /etc/postfix/mysql* all have fields user=, password=, hosts=, and dbname=. The queries in each of the files are as follows:
mysql_virtual_alias_maps.cf:query = SELECT goto FROM alias WHERE address='%s' AND active = 1
mysql_virtual_domains_maps.cf:query = SELECT domain FROM domain WHERE domain='%s'
mysql_virtual_mailbox_maps.cf:query = SELECT maildir FROM mailbox WHERE username='%s' AND active = 1
The /etc/courier/maildroprc file has the following contents:
# log the maildrop actions
logfile "/var/log/maildrop.log"
#
# parameters 1, 2, and 3 are the domain, user, and extension
DOMAIN=tolower("$1")
USER=tolower("$2")
EXTENSION=tolower("$3")
DEFAULT="/mail/$DOMAIN/$USER"
#
# try making a backup (cc) copy of the mail but do not abort if it fails
exception {
cc "$DEFAULT/.backup/"
}
#
# try delivering to the extension folder but do not abort if it fails
exception {
if(length("$EXTENSION") != 0 && "$EXTENSION" ne "backup")
{
to "$DEFAULT/.$EXTENSION/"
}
}
#
# deliver to the main inbox if there is no folder matching the extension or if no extension is specified
to "$DEFAULT/"
Installing Postfixadmin [1] was another challenge entirely. One of the complications of this is that there is no Debian package for Lenny (it seems that there will be one in Squeeze – Lenny+1).
I found David Goodwin’s tutorial on installing Postfixadmin and lots of other things on Debian/Etch [2] to be a very useful resource. I look forward to seeing a Lenny version of that document.
Please let me know if you can think of any way to improve this.
I am working on a system where a front-end mail server sends mail to what it considers to be a LDA (Local Delivery Agent) which actually sends mail to a back-end server via LMTP. I can’t remove that fake LDA from the design because it does a bunch of business specific processing along the way.
I am working on changing the back-end from Cyrus to Dovecot. Currently the mail goes from the fake LDA to the Cyrus LMTP server. What I would like to do is to have an LMTP server run on the back-end machine that launches the Dovecot deliver program immediately and then returns an appropriate code.
So far I have been experimenting with having Postfix run on the back-end to use deliver as the real LDA. The first problem with this is that the mail will be written to the Postfix queue and then written to the mail store. Doubling the number of writes is a real problem for a system that is going to be write-bottlenecked – it would significantly increase the hardware costs. The second problem is that when an account goes over quota the back-end server would be generating a bounce message. I would prefer the front-end server to generate the bounce on an un-munged message.
Basically all I need is a simple daemon (which could even be launched from inetd) that talks LMTP (a very simple cut-down version of SMTP) and executes a single command to receive the data. It might be necessary to serialise running the delivery process, in which case the mail data would have to be stored in memory and there would need to be a semaphore around executing the delivery program.
Does anyone know of such a program? If not then I’ll have to write it myself (which shouldn’t be difficult) and GPL it. If I have to do that then I need a suitable name for it. Any suggestions would be appreciated.
One of the recent poor trends in mailing list discussions is to reply to a message with a comment such as “FAIL” or “EPIC FAIL“.
The FAIL meme has been around for a while and actually does some good in some situations, slate has a good article about it [1]. The first example cited in that article is that ‘when Ben Bernanke and Henry Paulson testified before the Senate banking committee last month about Paulson’s proposed bailout bill, a demonstrator in the audience held up an 8.5-by-11 piece of paper with one word scrawled on it in block letters: “FAIL.”‘. This is an effective form of political demonstration, short words generally work well on placards (if only because the letters can be larger and therefore read from a greater distance) and anyone can understand the meaning of “FAIL” in that context.
There are some blogs dedicated to publicising supposed failures, failblog.org and ShipmentOfFail.com are two examples. I cite these as supposed failures because some of the pictures that they contain are obviously staged. It’s basically an Internet equivalent of the “Funniest Home Videos” shows that I never watched because they were not particularly funny.
So using the word “FAIL” on it’s own can be an effective form of political protest and can be used for mildly amusing web sites. But where it falls down is when it’s applied to a discussion that involves people who are from different cultures or have different levels of background knowledge – which covers most mailing list discussions.
Something that might be obviously wrong to some people is often not obvious at all to others. For example being forced to reboot a computer for any reason other than a kernel upgrade seems obviously wrong to me (and to most people who use Linux or other Unix systems) but Windows users seem happy to reboot machines after applying patches or upgrades. So writing a message with “FAIL” as the only word in a discussion with Windows users would not be productive. It could however be reasonable to forward a link to a page on a Microsoft web site to Linux people for their amusement with “FAIL” as the only comment – anyone who would find the link in question amusing would require no more explanation.
Sometimes when in a debate someone will write a message that only says “FAIL“, this is a very unconvincing argument that will not convince the opposition or any onlookers.
Generally it seems that using “FAIL” in a discussion with other like-minded people when talking about someone outside your group for the purpose of amusement can be effective. But any other use is going to be a “FAIL“.
As a more general rule single-word messages seem to have little value apart from certain limited situations. I have identified the following seven scenarios where a single word message is useful. Can anyone think of any others?
- Code review – someone posts code (or design for code) and people who like it will write “ACK” or something similar.
- Arranging a meeting – the question “who wants to meet for lunch tomorrow” has “me” as a valid answer.
- Voting – “yes” and “no” are valid answers for a poll, but a mailing list or forum probably isn’t the best place for it.
- Citing an example to refute a claim – often a single word won’t be a great response but may be adequate to prove a point.
- Answering a request for a recommendation – if asked to recommend a laptop I might say “Thinkpad” or if asked to recommend a server I might say “HP“. Both those answers are poor (I recommend EeePC for netbooks and Dell for small/cheap servers), so while such an answer would be useful it would be below my usual quality standards for email (I prefer to write at least two paragraphs explaining why I recommend something).
- Informing people that something has been done by replying to a request with the word “Done“.
- Agreeing to a contract or proposal with “OK” or “Yes“.
Update: I added another two reasonable uses of single word messages,
|
|
