Google Server Design

April 3, 2009 | etbe

Cnet has an article on the design of the Google servers [1]. It seems that their main servers are 2RU systems with a custom Gigabyte motherboard that takes only 12V DC input. The PSUs provide 12V DC and each system has a 12V battery backup to keep things running before a generator starts in the event of a power failure. They claim that they get better efficiency with small batteries local to the servers than with a single large battery array.

From inspecting the pictures it seems that the parts most likely to fail are attached by velcro. The battery is at one end, the PSU is at the other, and the hard disks are at one side. It appears that it might be possible to replace the PSU or the battery while the server is operational and in the rack.

The hard disks are separated from the motherboard by what appears to be a small sheet of aluminium which appears to give two paths for air to flow through the system. The thermal characteristics of the motherboard (CPUs) and the hard drives are quite different to having separate air flows seems likely to allow warmer air to be used in cooling the system (thus saving power).

Google boast that their energy efficiency now matches what the rest of the industry aims to do by 2011!

The servers are described as taking up 2RU, which gives a density of one CPU per RU. This surprised me as some companies such as Servers Direct [2] sell 1RU servers that have four CPUs (16 cores). Rackable systems [3] (which just bought the remains of SGI) sells 2RU half-depth systems (which can allow two systems in 2RU of rack space) that have four CPUs and 16 cores (again 4 CPUs per RU). Rackable systems also has a hardware offering designed for Cloud Computing servers, those CloudRack [4] systems have a number of 1RU trays. Each CloudRack tray can have as many as two server boards that has two CPUs (4 CPUs in 1RU) and 8 disks.

While I wouldn’t necessarily expect that Google would have the highest density of CPUs per rack, it did surprise me to see that they have 1/4 the CPU density of some commercial offerings and 1/8 the disk density! I wonder if this was a deliberate decision to use more server room space to allow slower movement of cooling air and thus save energy.

It’s interesting to note that Google have been awarded patents on some of their technology related to the batteries. Are there no journalists reading the new patents? Surely anyone who saw such patents awarded to Google could have published most of this news before Cnet got it.

Now, I wonder how long it will take for IBM, HP, and Dell to start copying some of these design features. Not that I expect them to start selling their systems by the shipping crate.

Censorship, Piracy, and Movie Ideas

March 31, 2009 | etbe

Flame has written a satirical post about the different methods used to try and prevent unauthorised use of copyright movies and the distribution of illegal porn [1]. He has also written an amusing rant about how offended he was by the false advertising of an erotic movie [2].

I think that both these issues need to be addressed at the same time. We need to have an erotic movie about a senator and some movie executives who go on a cruise. It could start with the senator wearing fishnet stockings under his suit while giving a speech about the need to prevent the distribution of pornography and “protect the children“. Then when on the the cruise he meets a young looking woman who likes wearing school uniforms and spends some of his spare time photographing her in artistic poses.

Most of the film would have gratuitous shots of people wearing less clothing than usual and spanking each other (including the senator giving a good impression of Dr Frank n Furter [3]). An element of mystery would also be good – who did the senator sleep with while blindfolded? The finale would have the senator in bed with the movie executives in an analogy of what senators do in real life (except that real life has a lot less spanking).

This plot wouldn’t necessarily imply a movie of Ed Wood [4] caliber. Let’s face it, most Hollywood movies don’t have a plot (the exceptions being those that are copied from successful movies from other times or places).

Disclaimer: This plot idea is strictly a work of fiction and bears no resemblance to any real people (I wouldn’t want my blog to end up in a secretive government blacklist). That said, if you want to imagine that any male politician who gives a speech about “protecting the children” is wearing lingerie while doing so, then go for it (you might even be right).

Inhaling Petrol Fumes

March 31, 2009 | etbe

Tonight at 6PM the channel 7 news will have a special report about the dangers of inhaling fumes while filling cars with petrol. I’m blogging now because based on past experience I expect almost no technical content in the report and that the advert for the news show contains everything that is useful.

The advert showed that they had used some film technique to show the fumes leaving the petrol tank while fuel is pumped in. It was obvious that the fumes were rising past the face of the man who was filling the tank. Not that this is really news, if you observe closely when fuel is pumped into a car you can see where the vapor escapes as the slight difference in density causes whatever is behind it to shimmer slightly.

For a long time the Prius has had a bladder inside the fuel tank which expands and contracts to match the fuel volume [1]. This greatly reduces the evaporative loss of fuel when the car is parked and the ambient temperature changes and also when fuel is pumped in. So it seems that in regard to the issue of fuel vapor poisoning the driver, the Prius has been better than other cars for about 10 years. Apparently Toyota have developed a new system that they have implemented in the Hybrid Lexus and the new Hybrid Camry which involves “vapor recovery”, this is supposed to give similar benefits in terms of not releasing petrol fumes into the environment but without limiting the capacity of the fuel tank in cold weather (a common complaint about the Prius). I have not been able to find any technical information on how this works (please let me know if you know a good web page about it).

The next issue is that any car which gives good fuel efficiency and has a reasonable size petrol tank will be better in regard to vapor releases. The less frequently you need to refuel your car the less fuel vapor you will inhale. I predict that channel 7 will not suggest that people drive fuel efficient cars, they are probably more likely to go for hand-wringing about the “inevitable” health problems, or make vague claims that the government should do something about it. The fact that the news show in question is immediately followed by “Today Tonight” (one of the tabloid TV shows) is an indication of the likely quality.

A final issue is the puddles of fuel that you commonly see at petrol stations. People regularly spill reasonable quantities of petrol and Diesel fuel and it just accumulates on the concrete. It’s not uncommon that I will have to drive with my car windows open after buying petrol due to the petrol that I stood in evaporating from my shoes. Apart from training people to not be stupid when refueling their vehicles I can’t think of any way of solving this problem. A petrol station employee once told me that it’s not uncommon for people to refuel cars with their young children standing within splash range of the fuel nozzle. If people can’t manage to avoid splashing their children with fuel then it seems that it will be impossible to get them to do anything reasonable or intelligent regarding the refueling process.

[1] http://www.vfaq.net/FAQ-fuel.html

Links March 2009

March 30, 2009 | etbe

Cory Doctorow has written a column for The Guardian titled “You shouldn’t have to sell your soul just to download some music” [1]. One really interesting point he made was “The same companies that spent decades telling lawmakers that they were explicitly not the guardians of the morality of the young that they couldn’t be held accountable for sex, drugs and rock’n’roll, for gangsta rap, for drug-fuelled dance-parties did a complete reversal and began to beat their chests about the corrupting influence of downloading on the poor kiddies“.

The pt4me2 organisation advocates that the Victorian government spend more money on public transport infrastructure [2]. Recently the congestion on Melbourne roads has been getting worse, the costs of private car parks in the central city area have been increasing, and during peak hours all public transport is over-crowded. Some significant improvements are needed, and more trains, trams, and buses are the only possible way of coping with the number of people working in the city. Also there’s the issue of suburbs that lack any public transport, I wouldn’t want to live near one of those areas when petrol hits $8/L (as the CSIRO predicts for 2018).

Michael Tiemann writes about Microsoft’s latest patent attack against Linux [3]. He doesn’t pull any punches which is a very noteworthy thing. It’s pretty rare to see someone in a senior position in a company blog in a way that makes any significant comment about another company.

Dan Ariely gave an interesting Ted.com talk about “Our Buggy Moral Code” [4]. One of the points was that people who declare themselves to be atheists are less likely to cheat when asked to swear on a bible first, another was that if someone who is identified as a member of the same group cheats then others will follow, but if an outsider cheats then they won’t. It seems that if you want to minimise cheating then you want to have only outsiders be seen to do it, and you want everyone to swear to follow some sort of moral code.

Flame has written a good rant titled “Anarchy is Dead” about the lack of popular outrage against the Australian government’s censorship plans [5].

Rolling Stone magazine has a good article about the US banking crisis [6]. A moderate amount of bad language, but no more than is deserved.

Amazing video of rounding up sheep that are fitted with LEDs at night [7].

Here’s an interesting TED talk by Saul Griffith about using kites to generate electricity [8]. Unfortunately he didn’t give much engineering information, it was more of an “executive summary” of what he has done. He claims that if a wartime level of effort was put in place then wind could supply all the electricity needs of the US within 10 years.

Adam Savage (of Mythbusters fame) gave a talk for TED about his obsession with the Dodo [9]. One thing that surprised me is how his manner in that talk differed from his appearance in Mythbusters. It seems that most of his crazy antics in Mythbusters are an act to entertain the audience. The information about the dodo was really interesting too. He then went on to talk about his recreation of the Maltese Falcon. His work can serve as inspiration for other geeks who want to try sculpting.

Linux Powered Battle Droids

March 29, 2009 | etbe

Flame has written some ideas about building Linux Powered Battle Droids and demonstrating them [1].

It seems to me that the biggest problem with having a battle of a similar nature to Robot Wars [2] is creating a safe arena. Getting an area the size of a basketball court fenced off with bullet-proof plastic is not going to be cheap.

The first solution to this problem that occurred to me was to have a battle held underwater. Water really slows down projectiles. The disadvantages of this are that it’s slightly more difficult to view. The viewing options are to using a web-cam, having a port-hole in the side of the pool (impossible if you don’t own the pool), and having a transparent viewing platform on the surface of the water (which would also be difficult and maybe expensive). Another disadvantage of submarine warfare is that having a droid spring a leak and quietly sink is not very dramatic.

It might be a better option to use aerial warfare via helicopters and balloons. The amount of weight that such craft can carry is not particularly great so there will be no real armor and fairly weak weapons. A motorbike helmet and a heavy coat should be all the protective equipment that is needed if the usual rules prohibiting projectile weapons from such contests are implemented.

A petrol powered model aeroplane could do some serious damage if it hit someone at full speed. But I don’t think that an indoor basketball court or any other enclosed arena that might be available would be large enough for fighter-planes. So it would be just balloons and helicopters.

Feeds and Banning from Planets

March 21, 2009 | etbe

Stewart Smith has written about the removal of a blog from Planet Linux Australia [1] due to publishing a list of URLs that the Australian government wants to censor.

The first point I want to make is that even if you had a list with thousands of entries that are not likely to offend anyone or incur any legal liability then it’s still not suitable for syndication on most Planet feeds. The correct thing to do is to have a paragraph describing the list and why people would want to read it and then use the MORE feature of your blog so that the rest isn’t in the RSS feed. If you use WordPress which seems to have the MORE function broken then that would mean hosting the list somewhere else.

In regard to the specific post, in a comment on Stewart’s post Matt suggests that the Planet software somehow filter out certain blog posts. I am not aware of any way of doing that apart from through code changes, Matt could submit some patches to allow that sort of thing.

One thing that would be really good would be to have an exclusion tag or category in a blog feed. So you for example you could have feed URLs such as /feed/lca which would be configured to list all posts without the tag not-lca. Another way for a blogger to do this would be to use Yahoo pipes [2]. The people who run a Planet should be prepared to take any feed URL. It would not be difficult for a blogger to create a pipe that excludes all items that have “NSFW” in the title (or any other possible way of listing them).

A final option is to have multiple blogs. I have a blog for documents that I regularly update [3]. Many of those documents had been plain HTML files edited with vi for years before I started blogging. But WordPress is a reasonable CMS and as I use it for blogging it made sense to use it for other documents too. WordPress has no good option for managing two types of documents, ones that are date-based (regular blog posts with the date in the URL) and non-date based (which change periodically and have different date stamps). There are WordPress pages, but the support for having moderate numbers of pages is not great. Also on my document blog I will often have articles appear new regularly as I change the date when updating them. Anyone is welcome to subscribe to the feed for my document blog if they are interested in seeing new versions of the documents, but I expect that most people don’t want to.

The Debian WordPress package (as of last time I used it) and my fork of the Debian WordPress package have great support for multiple blogs. There is WordPress-MU for bulk blog hosting, but that is only designed for people who want to run something like LiveJournal or Blogger. If you just want a few blogs for friends and relatives then the regular Debian WordPress package will do the job well.

Some bloggers maintain two blogs, one for public things and another for close friends and relatives (people who ARE interested in what they ate for breakfast). Having one blog for the NSFW material would be a reasonable thing to do for certain bloggers.

Finally while I doubt that someone who runs a Planet installation faces any legal liability, there is also the issue of a PR liability. From a PR perspective I think it’s best for the reputation of Linux users in Australia for certain things to not appear on Planet Linux Australia. That said it would be good if there was a process for removing and reinstating blogs that was publicly documented. There will obviously be many differences of opinion as to what is too risky to allow on the Planet so we should expect that from time to time feeds will be temporarily removed. When that happens what does a blogger have to do to be done to be syndicated again?

Update:

A comment has revealed a way of filtering out RSS feeds via the feed URLs used by wordpress. A URL such as /feed/cat=-X will give a feed of all articles that don’t contain category number X. Multiple categories can be specified when separated by commas. So this allows WordPress users to exclude their NSFW category from Planet Linux Australia.

Choosing a Server for CPU Intensive work

March 20, 2009 | etbe

A client is considering some options for serious deployment of some CPU intensive work. The options that are being considered include cloud computing (Amazon EC2 [1]), virtual machines (Slicehost [2] and Linode [3]), and purchasing servers to install in racks at various locations. I can’t disclose the criteria that will determine when each of those three options will be used (I expect that we will end up using all of them). But my research on the prices of various servers will hopefully be useful to someone.

For the server vendor I chose Dell. I believe that HP offers slightly better quality hardware than Dell, but they cost more and are more difficult to deal with (I can’t even get a price online). For this project I will be using a bunch of redundant servers (in a similar concept to the Google server array) so I’m not going to be overly bothered about losing a server occasionally – therefore the slight benefit that HP offers for reliability does not make up for the expense.

Dell has some 1RU servers that have two CPU sockets and allow eight CPU cores. It seems that the best value that Dell offers for a server without RAID (the entire server is redundant) is a PowerEdge SC1435 that has two Opteron 2352 quad-core CPUs running at 2.1GHz, 4G of RAM, a 1TB SATA disk, and a Broadcom PCIe Gig-e card for $3,816.50. That machine gives an option of 2.3GHz CPUs for an extra $621.50, I am not sure that increasing the clock speed by almost 10% for a 16% increase in system price is a good idea.

The second best option was a PowerEdge 1950 III that has two Xeon E5420 2.5GHz quad-core CPUs with 12M of cache, 4G of RAM and a 1TB SATA disk for $4,302.30. The Intel option has 3 years of support included while the AMD option included 1 year of support and needed at least an extra $990 for 3 years of support. So it seems that if 3 years of support is desired then the Intel based server becomes significantly cheaper and is probably a better option.

Dell’s 2RU and 4RU servers are of no interest if you want CPU performance. The 2RU servers only support two processors and the 4RU servers only support four processors. So it’s a ratio of 2 processors per RU for 1RU servers vs one processor per RU for 2RU and 4RU servers, and the 2RU and 4RU servers are a lot more expensive too.

I am investigating the Dell blade server. Blade servers are great for CPU density and good for management. The Dell blade enclosure M1000e takes 10RU of space and supports 16 half-height blades or 8 full-height blades. The Dell M905 blade supports four AMD quad-core processors for a total of 128 cores in 10RU, there are also half-height blades that support two quad-core processors for the same CPU density.

So in terms of CPU density it’s an average of 12.8 cores per RU for the blade server vs 8 cores per RU for 1RU servers. While I haven’t got a complete price yet, it seems that four CPUs suitable for the M905 will cost about as much as four 1RU servers. So the 1RU systems are definitely better value for money than the blade server. The difference is the management cost. N servers that have two CPUs will be more work than N/2 servers that have four CPUs, but on the other hand blade servers require some specialised skills to run them (which I don’t have) and that might also cause problems. I don’t think that blades will be part of this project.

Dear Magazines – Please Publish Your Adverts Online

March 20, 2009 | etbe

When reading a magazine I often see an advert for a product that I want to buy (or recommend that a client buy). This is of course expected as the advertisers put a lot of effort into targeting their adverts to the people who read such magazines. However I often decide that I want to buy the product some weeks after reading the magazine.

Linux Journal [1] usually has adverts for good server hardware that supports Linux. Their web site is quite nice in many ways, it allows subscribers to read articles online and has an index to all back-issues. But it has no link that I could find for reading the adverts! I would like to see an index of all advertising that has ever been published so that I can read the adverts online. The ancient adverts are good for historical reference and the new ones are good for purchasing decisions.

Right now I would like to be suggesting that a client consider buying a number of servers from a company that advertises in Linux Journal, but I’m not because they don’t publish their adverts online!

To make matters worse the Linux Journal web site doesn’t include a suitable contact address for issues unrelated to subscriptions. I believe that a magazine needs to publish an email address for copyright infringement reports (I often notify magazines when someone rips off their content so that they can issue a DMCA take-down notice), an email address for technical problems (I often notice rendering errors and broken links in web pages and like to report them), and an email address for random stuff. I’m even happy to use a web-based form to submit my suggestions if that’s what makes them happy, but having no published way of notifying them is simply a mistake.

NB I do have some email addresses of LJ employees in my addressbook somewhere, I will send them email if no-one responds to this blog post. But I’m publishing this because it’s probably a mistake that other magazines make, and because any response from them is going to be too late for the report on hardware prices that I’m writing.

Update: Linux Journal does allow you to download a sample copy, that is a PDF of one complete issue that includes adverts. So I can see adverts from August 2008.

[1] http://www.linuxjournal.com

The Cost of Car Crashes

March 19, 2009 | etbe

An article from 1999 suggested that car crashes caused a financial loss in OECD countries equivalent of 2% of their entire economies [1]. An article from the Sydney Morning Herald in 2001 gave a conservative estimate of the cost of a road fatality at $1.5 million [2], it also notes that due to different analysis methods American transport economists derived a figure of $5.5 million. $1.5 million in 2001 adjusted for CPI would be close to $2 million now.

Currently that $2M cost is an externality of the car industry. Most of it is paid by the government, IE we all pay for it through our taxes. This means that there is little financial incentive for drivers and car companies to make the roads safer. Many of the attempts to legislate road safety fail due to the legal system being unable to manage the rapidly changing range of vehicles on the market.

The insurance companies have very detailed analysis of the relative safety of vehicles, so it seems that the only sensible way of enforcing safe driving is through economic measures implemented via insurance.

I believe that for every person who is killed or seriously injured on the road a fine of $2M should be levied. Every driver should be compelled to have insurance to cover such fines (driving without insurance should be illegal).

Then the government could cease being involved in regulating what types of car someone can drive. If someone who is less than 25 years old can get insurance for a turbo-charged car then it probably means that a statistical analysis suggests that the combination of driver and vehicle is likely to be reasonably safe (EG there are many turbo-charged cars on the market that are not particularly fast).

Now this will increase the car insurance costs for everyone, but it will decrease the amount of general tax money that is spent on issues related to road fatalities, which would allow the income tax rates to be decreased. This means that any tax-payer who has a good driving record and who drives a type of car that tends not to be crashed could expect to save money overall. Any tax-payer who doesn’t drive a car would save even more money.

But the main point of this idea is to increase road safety by forcing bad cars and drivers off the road. Currently defective cars are only removed from the road if police notice something unsafe about them and cite them for being unroadworthy – this only happens if it’s a problem which can be observed from outside the vehicle (EG worn tires or broken lights). In some states elderly drivers have no requirement for periodic health checks to determine their ability to drive, I know of one case of a woman who was certified as legally blind, ordered a white cane, and then drove home afterwards! I’m sure that insurance companies would implement whatever tests are necessary to reduce the risk of being hit by multiple $2M fines from a single crash.

Hyperthermia and Children in Cars

March 18, 2009 | etbe

Bruce Schneier writes about the risks involving children abandoned in cars and cites an article about the tragic deaths of children in hot cars [1]. One unfortunate error that he made was to not cite the following from the end of the last page of the Washington post article he cited [2]:
In hyperthermia cases, he believes, the parents are demonized for much the same reasons. “We are vulnerable, but we don’t want to be reminded of that. We want to believe that the world is understandable and controllable and unthreatening, that if we follow the rules, we’ll be okay. So, when this kind of thing happens to other people, we need to put them in a different category from us. We don’t want to resemble them, and the fact that we might is too terrifying to deal with. So, they have to be monsters.”

I believe that similar thought processes are used in relation to many other situations, and that such thought processes prevent people from taking appropriate actions to minimise the risk. If someone considers that forgetting a child in the back seat to be an accident that could happen to anyone then they would be inclined to take action to minimise the risk (such as spending some money on a sensor). If however they consider such forgetfulness to be proof of being a “bad parent”, then as they are a “good parent” they would have to avoid buying a monitor. I’m surprised that Bruce didn’t draw an analogy between this and the forgetful losses of laptops and guns by people who work for law enforcement agencies (which he has written about before).

I wonder how expensive it would be to make a sensor for heart-rate, breathing, and temperature integrated with a GSM modem and a GPS? If it could be small enough to be attached to clothes then the child could wear it at all times.

If such a sensor was to detect a sign of a problem it wouldn’t matter whether the child was forgotten in a car, at day-care, or even being actively supervised. The data would be sent to the monitoring agency along with GPS data. The monitoring agency could then phone the parents. If the parents don’t answer or don’t know where the child is then the police could track down the GPS location. Probably most calls would be due to parents leaving a child too close to an air-conditioner or playing outside in the sun in summer which are unlikely to give a fatal result and a phone call would get a quick fix for what would only be a minor health problem.

If the device was marketed as monitoring for “sleep apnia” then parents could buy it without admitting to the possibility that they might do anything wrong. The causes of SIDS are a topic of ongoing research and parents can admit to being worried about their children suffering from it without admitting any possibility that they might make a mistake.

etbe – Russell Coker

Linux, politics, and other interesting things