Terrorism Foolishness

July 7, 2007 | etbe

The Age has published a remarkably stupid article about terrorism titled “It’s hard to prevent the hard to imagine” which contains some amusing statements such as “a plan to use liquid explosives hidden in soft-drink bottles and destroy commercial jets crossing the Atlantic. The scale of this plot, combined with the innovative bomb design, threatened to kill thousands of people and cause massive disruption to global commerce“. This however has been debunked by many chemists, here is one of the many expert analysis of the claims.

Here’s another inane claim from The Age “So the perpetrators in the UK looked elsewhere and compiled a crude yet potentially deadly bomb from materials available in everyday life â€” a mix of gas cylinders, petrol and nails. Finding a way to govern access to such otherwise mundane items will be expensive, and perhaps ultimately, impossible.“. It’s surprising that The Age editorial team were unable to find someone who knows the basics of chemistry or who has ever had a need to start a fire to review the article. Everyone should know that an oxidising agent is necessary for combustion – and a huge quantity of an oxidising agent is needed for an explosion. Petrol vapour won’t ignite if it’s too concentrated (it can displace enough oxygen to prevent combustion). When a gas cylinder is damaged and set alight you get a fireball surrounding it which may be several meters wide (the one occasion that I witnessed a damaged gas cylinder on fire the fire-ball was about 4 meters wide). But no explosion. To propell nails or other solid objects you need the combustion to be rapid and only on one side of the object. Combustion on all sides (IE a 4 meter wide fire-ball) will not propell nails. Here’s what a British Army bomb-disposal operator has to say about it in The Register.

If you want to talk about dangerous items that are difficult to control how about cars? A terrorist who drove a 4WD along the footpath of Oxford St could potentially kill more people than the London bombings and would take almost no preparation.

The article inevitably concludes with claims about the efforts that al Qaeda operatives are supposedly making to recruit people for terrorist missions. Naturally it ignores the best recruiting method of al Qaeda – the huge civilian death toll in Iraq since the US led invasion. The best available medical research (by researchers from Johns Hopkins University and published in The Lancet – the most reputable and prestigious medical journal) estimates that there were 655,000 “excess deaths” as a result of the invasion in the period up to July 2006. Over the last year the best available reports suggest that the violence in Iraq has got worse (among other things the number of US air-strikes is twice what it was last year).

For more analysis of the fear-mongering being done by much of the media (including The Age) here’s another article from The Register.

It’s interesting to read The Age’s article Truth first casualty of the internet?. Maybe they should publish an article titled Intelligence first casualty of print media?.

Committing Data to Disk

July 2, 2007 | etbe

I’ve just watched the video of Stewart Smith’s LCA talk Eat My Data about writing applications to store data reliably and not lose it. The reason I watched it was not to learn about how to correctly write such programs, but so that I could recommend it to other people.

Recently I have had problems with a system (that I won’t name) which used fwrite() to write data to disk and then used fflush() to commit it! Below is a section from the fflush(3) man page:

NOTES
       Note that fflush() only flushes the user space buffers provided by  the
       C  library.   To  ensure that the data is physically stored on disk the
       kernel buffers must be flushed too, e.g. with sync(2) or fsync(2).

Does no-one read the man pages for library calls that they use?

Then recently I discovered (after losing some data) that both dpkg and rpm do not call fsync() after writing package files to disk. The vast majority of Linux systems use either dpkg or rpm to manage their packages. All those systems are vulnerable to data loss if the power fails, a cluster STONITH event occurs, or any other unexpected reboot happens shortly after a package is installed. This means that you can use the distribution defined interface for installing a package, be told that the package was successfully installed, have a crash or power failure, and then find that only some parts of the package were installed. So far I have agreement from Jeff Johnson that RPM 5 will use fsync(), no agreement from Debian people that this would be a good idea, and I have not yet reported it as a bug in SUSE and Red Hat (I’d rather get it fixed upstream first).

During his talk Stewart says sarcastically “everyone uses the same filesystem because it’s the one true way“. Unfortunately I’m getting this reaction from many people when reporting data consistency issues that arise on XFS. The fact that Ext3 by default will delay writes by up to 5 seconds for performance (which can be changed by a mount option) and that XFS will default to delaying up to 30 seconds means that some race conditions will be more likely to occur on XFS than in the default configuration of Ext3. This doesn’t mean that they won’t occur on Ext3, and certainly doesn’t mean that you can rely on such programs working on Ext3.

Ext3 does however have the data=ordered mount option (which seems to be the default configuration on Debian and on Red Hat systems), this means that meta-data is committed to disk after the data blocks that it referrs to. This means that an operation of writing to a temporary file and then renaming it should give the desired result. Of course it’s bad luck for dpkg and rpm users who use Ext3 but decided to use data=writeback as they get better performance but significantly less reliability.

Also we have to consider the range of filesystems that may be used. Debian supports Linux and HURD kernels as main projects and there are less supported sub-projects for the NetBSD, FreeBSD, and OpenBSD kernels as well as Solaris. Each of these kernels has different implementations of the filesystems that are in common and some have native filesystems that are not supported on Linux at all. It is not reasonable to assume that all of these filesystems have the same caching algorithms as Ext3 or that they are unlike XFS. The RPM tool is mainly known for being used on Red Hat distributions (Fedora and RHEL) and on SuSE – these distributions include support for Ext2/3, ReiserFS, and XFS as root filesystems. RPM is also used on BSD Unix and on other platforms that have different filesystems and different caching algorithms.

One objection that was made to using fsync() was the fact that cheap and nasty hard drives have write-back caches that are volatile (their contents dissappear on power loss). As with such drives reliable operation will be impossible so why not just give up! Pity about the people with good hard drives that don’t do such foolishness, maybe they are expected to lose data as an expression of solidarity with people who buy the cheap and nasty hardware.

Package installation would be expected to be slower if all files are sync’d. One method of mitigating this is to write a large number of files (EG up to a maximum of 900) and then call fsync() on each of them in a loop. After the last file has been written the first file may have been entirely committed to disk, and calling fsync() on one file may result in other files being synchronised too. Another issue is that the only time package installation speed really matters is during an initial OS install. It should not be difficult to provide an option to not call fsync() for use during the OS install (where any error would result in aborting the install anyway).

Update: If you are interested in disk performance then you might want to visit the Benchmark category of my blog, my Bonnie++ storage benchmark and my Postal mail server benchmark.

Update: This is the most popular post I’ve written so far. I would appreciate some comments about what you are interested in so I can write more posts that get such interest. Also please see the Future Posts page for any other general suggestions.

Xen and Heartbeat

June 24, 2007 | etbe

Xen (a system for running multiple virtual Linux machines) and has some obvious benefits for testing Heartbeat (the clustering system) – the cheapest new machine that is on sale in Australia can be used to simulate a four node cluster. I’m not sure whether there is any production use for a cluster running under Xen (I look forward to seeing some comments or other blog posts about this).

Most cluster operations run on a Xen virtual machine in the same way as they would under physically separate machines, and Xen even supports simulating a SAN or fiber-channel shared storage device if you use the syntax phy:/dev/VG/LV,hdd,w! in the Xen disk configuration line (the exclamation mark means that the volume is writable even if someone else is writing to it).

The one missing feature is the ability to STONITH a failed node. This is quite critical as the design of Heartbeat is that a service on a node which is not communicating will not be started on another node until the failed node comes up after a reboot or the STONITH sub-system states that it has rebooted it or turned it off. This means that the failure of a node implies the permanent failure of all services on it until/unless the node can be STONITH’d.

To solve this problem I have written a quick Xen STONITH module. The first issue is how to communicate between the DomU’s (Xen virtual machines) and the Dom0 (the physical host). It seemed that the best way to do this is to ssh to special accounts on the Dom0 and then use sudo to run a script that calls the Xen xm utility to actually restart the node. That way the Xen virtual machine gets limited access to the Dom0, and the shell script could even be written to allow each VM to only manage a sub-set of the VMs on the host (so you could have multiple virtual clusters on the one physical host and prevent them from messing with each other through accident or malice).

xen ALL=NOPASSWD:/usr/local/sbin/xen-stonith

Above is the relevant section from my /etc/sudoers file. It allows user xen to execute the script /usr/local/sbin/xen-stonith as root to do the work.

One thing to note is that from each of the DomU’s you must be able to ssh from root on the node to the specified account for the Xen STONITH service without using a password and without any unreasonable delay (IE put UseDNS no in /etc/ssh/sshd_config.

The below section (which isn’t in the feed) there are complete scripts for configuring this.

Continue reading →

Pointers in Bash Shell Scripts

June 18, 2007 | etbe

# foo=bar
# name=foo
# echo ${!name}
bar

The above example shows how to make a bash environment variable reference the value of another.

# echo ${!f*}
foo fs_abc_opts fs_name

The above example shows how to get a list of all variables beginning with “f” (NB the wildcard can only go at the end).

A Lack of Understanding of Nuclear Issues

June 17, 2007 | etbe

Ben Fowler writes about the issues related to nuclear power in Australia. He spends 8 paragraphs discussing the issues on the “Right” side of politics – of which 6 concern the an Australian nuclear weapons capability and then spends 3 out of 5 paragraphs related to the “Left” side explaining that he thinks that everyone who opposes nuclear power is a Luddite.

Ben didn’t bother providing any links or statistics to back up his claims, so I’ll assist him in analysing these issues by providing some facts that we can discuss.

In March Spain had wind power provide 27% of all electricity (making wind power the main source of power for the country). I blogged about this at the time. While Spain has an ongoing program of building new wind power stations the majority of wind turbines in Spain are quite old (the Model T of wind power) and not nearly as efficient as modern turbines that would be installed for Australian power plants.

The Danish government has announced plans to use wind power for 75% of their electricity. Denmark has a much smaller land area than Australia, which means that generating so much electricity from wind power is more technically challenging for them than it would be for us. A larger land area means that when one area has low wind speeds other areas can be used to provide power.

For home electricity generation wind turbines have not yet been proven to be practical. The linear speed of the blade is determined by the wind speed and the rotational speed is therefore a factor of the wind speed divided by the radius of the blades. This means that smaller turbines have higher rotational speeds which causes more noise (bad for getting council approval), also to avoid turbulence a wind turbine will ideally be some distance above the ground (8 meters is good) which again gives problems when getting approval. The O’Connor Hush Turbine is supposed to solve the noise component of this problem. It will be interesting to see whether home based wind power becomes practical in future – if so I would like to get an O’Connor turbine on my house!

Home solar power has been proven to work well, in the form of both solar-electric and solar hot water (I know several people who have been happily using them for years). You don’t get cold showers when the sun isn’t shining, you instead use gas or electricity to heat the water (it’s a standard feature in a solar hot water system). Also your home electricity doesn’t go off when the sun stops shining, you have batteries that are charged during sunny times to run things overnight, and when they get flat you pay for power from the grid.

It is quite realistic to stick solar power systems on every roof in the country. The added cost to the process of building or purchasing a house is negligible and the benefits include having electricity when mains power is unavailable (NB water is used in generating electricity from coal or nuclear power plants so a bad drought will eventually become a time of limited mains power). Even the smallest home solar electric system will produce enough electricity to power a fridge and freezer 24*7 so it’s a very useful backup for essential power. The government is subsidising the installation of solar electric systems, so it seems that they expect everyone to get one eventually.

Dr. Ziggy Switkowski (the main advocate of nuclear power in Australia) says “the introduction of a carbon tax could make nuclear power the cheapest option by the 2020s”. In consecutive paragraphs Ben derides “carbon trading” and claims that nuclear power is “practical”. Unfortunately the main advocate of nuclear power in Australia does not believe that it is practical without a carbon tax. Ziggy also states that it would take at least 15 years to complete a nuclear power plant, unfortunately we don’t have the luxury of waiting for 15 years before starting to try and reduce the damage that we are doing to the environment. The Stern report makes the economic consequences of inaction quite clear.

I am not a Luddite. I oppose nuclear power because of the risks related to accidental release of radioactive material and the routine release of radioactive material as part of the uranium mining process, and the dangers related to long-term storage of nuclear waste (let’s not assume that Star Trek science can make it all non-radioactive). Nuclear power is not cost effective for Australia and will take so long to develop that it won’t save us from the serious economic damage predicted by the best scientific models as presented in the Stern report.

For large scale power generation wind power works now, can be easily implemented and has no hidden costs or risks. There will never be a Chernobyl type accident with wind power, it is inherently a safe technology. For small scale power generation (something you can add to your home) solar power works well, is not expensive (when considering the price of a house and especially when the government subsidy is counted) and has the potential to seriously reduce the amount of carbon dioxide produced.

Easy Search Engine Optimisation in WordPress

June 11, 2007 | etbe

The first thing you need is to have meta keywords in the pages. The solution I discovered for doing this is the All In One SEO Pack, this automatically makes the categories of a blog post become meta keywords, allows adding arbitrary meta keywords when editing the blog post, and supports keywords for the entire blog. It also re-writes the titles to have the title of the post appear before the title of the blog in the <TITLE> tag which is rumored to be a good thing to do.

The next thing you want to do is to create a Google sitemap. A sitemap is an XML file that describes all the URLs in your web site which tells Google (and other search engines) the last change dates of all the URLs and the change frequencies (or how often they should be checked for changes). Note that the format is called Google Sitemaps because Google invented it, other web search engines also use the same format. To generate my sitemap I use the Google Sitemaps Generator which tells earch engines to index my main blog page daily and the category pages weekly (which is configurable). It also has an option to notify Google of changes by pinging the Google service.

Having multiple names for the same web page hurts you when it comes to search engines. Having two separate pages at the top of page 2 is not nearly as good as having one page on page one! To solve this problem I use the Permalink Redirect plugin to make sure that each page has only one URL pointing to it. This plugin also allows redirecting the feed URL to feedburner.com. Feedburner serves the feed data (thus saving you some bandwidth if you are on a slow link) and also tracks statistics on it which are interesting.

Under the Writing section of the Options menu there is an option to enter a list of URLs for update services that can be notified of every post. I use http://rpc.pingomatic.com/ and http://rpc.technorati.com/rpc/ping. The Update Services entry in the WordPress Codex has more information on how this works.

Social networking sites such as digg.com can direct a large amount of traffic to your site if you write about a popular topic. The Share This plugin is the simplest way I found of adding social networking hooks to WordPress (I tried a few others that had both less functionality and less ease of use).

Finally before publishing a post, I often check the Google Adwords Keyword Tool. This often gives me ideas on how to make the meta tags and title match what people are searching for.

configuring a Heartbeat service

June 9, 2007 | etbe

In my last post about Heartbeat I gave an example of a script to start and stop a cluster service. In that post I omitted to mention that the script goes in the directory /usr/lib/ocf/resource.d/heartbeat.

To actually use the script you need to write some XML configuration to tell Heartbeat which parameters should be passed to it via environment variables and which nodes may be candidates to run it.

In the below example the type of web means that the script /usr/lib/ocf/resource.d/heartbeat/web will be called to do the work. The id attributes are all arbitrary, but you want to decide on some sort of consistent naming scheme. I have decided to name web server instances web-X where X is the IP address used for providing the service.

The nvpair element contains a configuration option that will be passed to the script as an environment variable. The name of ip means that the environment variable will be named OCF_RESKEY_ip. Naming of such variables is arbitrary and a script may take many variables. A well written script (which incidentally does not mean my previous blog post) will have an option meta-data to give XML output describing all the variables that it accepts. An example of this can be seen by the command /usr/lib/ocf/resource.d/heartbeat/IPaddr2 meta-data.

In the XML the resources section (as specified by --obj_type resources on the cibadmin command-line) describes resources that the Heartbeat system will run. The constraints section specifies a set of rules that determine where they will run. If the symmetric-cluster attribute in the cluster_property_set is set to true then resources will be permitted to run anywhere, if it is set to false then a resource will not run anywhere unless there is a constraint specifying that it should do so – which means that there must be at least one constraint rule for every resource that is permitted to run.

In the below example I have constraint rules for the service giving node-0 and node-1 a priority of 9000 for running the service.

In a future post I will describe the cluster_property_set and how it affects calculations of where resources should run.
Continue reading →

Heartbeat service scripts

June 8, 2007 | etbe

A service script for Heartbeat needs to support at least three operations, start, stop, and status. The operations will return 0 on success, 7 on failure (which in the case of the monitor script means that the service is not running) and any other value to indicate that something has gone wrong.

In the second half of this post (not in the feed) I have included an example service script. It is a very brief script and does not support some of the optional parameters (monitor, validate-all, and meta-data). So this script is not of a quality that would be accepted for inclusion in a Heartbeat release but is adequate to demonstrate the concepts.

The XML configuration for a service can have an arbitrary set of name-value pairs, and they are passed to the script as environment variables. For example the below script expects that the XML configuration item named ip will have the IP address used by the service, my script receives this as a variable named OCF_RESKEY_ip. My script doesn’t use the address, it merely allows it to be inherited by the IPaddr2 script (which is part of the Heartbeat distribution) and that script assigns the address to an Ethernet interface.

The script is for testing Heartbeat, it mounts a filesystem and starts Apache (which is configured to serve web pages from the filesystem in question on the IP address supplied by the ip parameter).

For test purposes the script looks for a file named /root/fail, if this file exists then the status check will always abort. An aborting status script means that Heartbeat can not be certain that the node in question has released all resources that it was using for the service. This means that Heartbeat will have to kill the node via the STONITH service. Such test scripts are the only way to test that STONITH works, and I believe that it’s a necessary part of pre-production testing of a Heartbeat cluster.

Update: Made it display error messages in all cases and also reformatted it for better cut/paste.
Continue reading →

Another Heartbeat 2.0 STONITH example configuration

May 30, 2007 | etbe

In a Heartbeat cluster installation it may not be possible to have one STONITH device be used to reboot all nodes. To support this it is possible to have multiple STONITH devices configured that will each be used to reboot different nodes in the cluster. In the following code section there is an example of how to configure STONITH for two separate ssh instances. Of course this is not useful apart from as an example of how to configure STONITH. It would be quite easy to change one of those ssh configuration entries to use IPMI or some more effective method of managing machines. My previous post on this topic has an example of a simpler ssh STONITH configuration.

It is convenient that the ssh method for rebooting nodes is available both as a shared object (which is used by the following example XML) and as a shell script (type external/ssh). The shell script can be used to give the same functionality as the shared object (with reduced performance) but the benefit is as an example of how to write external plugins. For a client I have just written an IPMI module that works with machines that have two Ethernet ports. When a server has two Ethernet ports you want to send an IPMI reset command to both of them in case the failure which requires a STONITH was triggered by a broken Ethernet cable. Unfortunately I can’t release the IPMI code at this time

Continue reading →

Heartbeat version 2.0 CIB STONITH example configuration

May 29, 2007 | etbe

Below is a sample script to configure the ssh STONITH agent for the Heartbeat system. STONITH will reboot nodes when things go wrong to restore the integrity of the cluster.

The STONITH test program supports the -n option to list parameters and the -l option to list nodes. The following is an example of using it with the ssh agent:
# stonith -t ssh -n hostlist # stonith -t ssh hostlist="node-0 node-1" -l node-0 node-1

The hostlist tuple is the only configuration option for ssh. It is assumed that you have passwordless ssh logins allowed between root on all the nodes in the cluster so the host name list is all that’s needed.

The important thing to note about the constraint is that you are constraining the parent of the clones (which in this example has an ID of “DoFencing“) not a clone instance.

ssh is the simplest and in many ways least useful method of STONITH, but it’s good for an example as everyone knows it. Once you get ssh going it’ll be trivial to get the other methods working.

See below for the script to insert the XML in the CIB.
Continue reading →

etbe – Russell Coker

Linux, politics, and other interesting things

Tag Archives: Best Posts