Terrorism Foolishness

July 7, 2007 | etbe

The Age has published a remarkably stupid article about terrorism titled “It’s hard to prevent the hard to imagine” which contains some amusing statements such as “a plan to use liquid explosives hidden in soft-drink bottles and destroy commercial jets crossing the Atlantic. The scale of this plot, combined with the innovative bomb design, threatened to kill thousands of people and cause massive disruption to global commerce“. This however has been debunked by many chemists, here is one of the many expert analysis of the claims.

Here’s another inane claim from The Age “So the perpetrators in the UK looked elsewhere and compiled a crude yet potentially deadly bomb from materials available in everyday life â€” a mix of gas cylinders, petrol and nails. Finding a way to govern access to such otherwise mundane items will be expensive, and perhaps ultimately, impossible.“. It’s surprising that The Age editorial team were unable to find someone who knows the basics of chemistry or who has ever had a need to start a fire to review the article. Everyone should know that an oxidising agent is necessary for combustion – and a huge quantity of an oxidising agent is needed for an explosion. Petrol vapour won’t ignite if it’s too concentrated (it can displace enough oxygen to prevent combustion). When a gas cylinder is damaged and set alight you get a fireball surrounding it which may be several meters wide (the one occasion that I witnessed a damaged gas cylinder on fire the fire-ball was about 4 meters wide). But no explosion. To propell nails or other solid objects you need the combustion to be rapid and only on one side of the object. Combustion on all sides (IE a 4 meter wide fire-ball) will not propell nails. Here’s what a British Army bomb-disposal operator has to say about it in The Register.

If you want to talk about dangerous items that are difficult to control how about cars? A terrorist who drove a 4WD along the footpath of Oxford St could potentially kill more people than the London bombings and would take almost no preparation.

The article inevitably concludes with claims about the efforts that al Qaeda operatives are supposedly making to recruit people for terrorist missions. Naturally it ignores the best recruiting method of al Qaeda – the huge civilian death toll in Iraq since the US led invasion. The best available medical research (by researchers from Johns Hopkins University and published in The Lancet – the most reputable and prestigious medical journal) estimates that there were 655,000 “excess deaths” as a result of the invasion in the period up to July 2006. Over the last year the best available reports suggest that the violence in Iraq has got worse (among other things the number of US air-strikes is twice what it was last year).

For more analysis of the fear-mongering being done by much of the media (including The Age) here’s another article from The Register.

It’s interesting to read The Age’s article Truth first casualty of the internet?. Maybe they should publish an article titled Intelligence first casualty of print media?.

Committing Data to Disk

July 2, 2007 | etbe

I’ve just watched the video of Stewart Smith’s LCA talk Eat My Data about writing applications to store data reliably and not lose it. The reason I watched it was not to learn about how to correctly write such programs, but so that I could recommend it to other people.

Recently I have had problems with a system (that I won’t name) which used fwrite() to write data to disk and then used fflush() to commit it! Below is a section from the fflush(3) man page:

NOTES
       Note that fflush() only flushes the user space buffers provided by  the
       C  library.   To  ensure that the data is physically stored on disk the
       kernel buffers must be flushed too, e.g. with sync(2) or fsync(2).

Does no-one read the man pages for library calls that they use?

Then recently I discovered (after losing some data) that both dpkg and rpm do not call fsync() after writing package files to disk. The vast majority of Linux systems use either dpkg or rpm to manage their packages. All those systems are vulnerable to data loss if the power fails, a cluster STONITH event occurs, or any other unexpected reboot happens shortly after a package is installed. This means that you can use the distribution defined interface for installing a package, be told that the package was successfully installed, have a crash or power failure, and then find that only some parts of the package were installed. So far I have agreement from Jeff Johnson that RPM 5 will use fsync(), no agreement from Debian people that this would be a good idea, and I have not yet reported it as a bug in SUSE and Red Hat (I’d rather get it fixed upstream first).

During his talk Stewart says sarcastically “everyone uses the same filesystem because it’s the one true way“. Unfortunately I’m getting this reaction from many people when reporting data consistency issues that arise on XFS. The fact that Ext3 by default will delay writes by up to 5 seconds for performance (which can be changed by a mount option) and that XFS will default to delaying up to 30 seconds means that some race conditions will be more likely to occur on XFS than in the default configuration of Ext3. This doesn’t mean that they won’t occur on Ext3, and certainly doesn’t mean that you can rely on such programs working on Ext3.

Ext3 does however have the data=ordered mount option (which seems to be the default configuration on Debian and on Red Hat systems), this means that meta-data is committed to disk after the data blocks that it referrs to. This means that an operation of writing to a temporary file and then renaming it should give the desired result. Of course it’s bad luck for dpkg and rpm users who use Ext3 but decided to use data=writeback as they get better performance but significantly less reliability.

Also we have to consider the range of filesystems that may be used. Debian supports Linux and HURD kernels as main projects and there are less supported sub-projects for the NetBSD, FreeBSD, and OpenBSD kernels as well as Solaris. Each of these kernels has different implementations of the filesystems that are in common and some have native filesystems that are not supported on Linux at all. It is not reasonable to assume that all of these filesystems have the same caching algorithms as Ext3 or that they are unlike XFS. The RPM tool is mainly known for being used on Red Hat distributions (Fedora and RHEL) and on SuSE – these distributions include support for Ext2/3, ReiserFS, and XFS as root filesystems. RPM is also used on BSD Unix and on other platforms that have different filesystems and different caching algorithms.

One objection that was made to using fsync() was the fact that cheap and nasty hard drives have write-back caches that are volatile (their contents dissappear on power loss). As with such drives reliable operation will be impossible so why not just give up! Pity about the people with good hard drives that don’t do such foolishness, maybe they are expected to lose data as an expression of solidarity with people who buy the cheap and nasty hardware.

Package installation would be expected to be slower if all files are sync’d. One method of mitigating this is to write a large number of files (EG up to a maximum of 900) and then call fsync() on each of them in a loop. After the last file has been written the first file may have been entirely committed to disk, and calling fsync() on one file may result in other files being synchronised too. Another issue is that the only time package installation speed really matters is during an initial OS install. It should not be difficult to provide an option to not call fsync() for use during the OS install (where any error would result in aborting the install anyway).

Update: If you are interested in disk performance then you might want to visit the Benchmark category of my blog, my Bonnie++ storage benchmark and my Postal mail server benchmark.

Update: This is the most popular post I’ve written so far. I would appreciate some comments about what you are interested in so I can write more posts that get such interest. Also please see the Future Posts page for any other general suggestions.

Xen and Heartbeat

June 24, 2007 | etbe

Xen (a system for running multiple virtual Linux machines) and has some obvious benefits for testing Heartbeat (the clustering system) – the cheapest new machine that is on sale in Australia can be used to simulate a four node cluster. I’m not sure whether there is any production use for a cluster running under Xen (I look forward to seeing some comments or other blog posts about this).

Most cluster operations run on a Xen virtual machine in the same way as they would under physically separate machines, and Xen even supports simulating a SAN or fiber-channel shared storage device if you use the syntax phy:/dev/VG/LV,hdd,w! in the Xen disk configuration line (the exclamation mark means that the volume is writable even if someone else is writing to it).

The one missing feature is the ability to STONITH a failed node. This is quite critical as the design of Heartbeat is that a service on a node which is not communicating will not be started on another node until the failed node comes up after a reboot or the STONITH sub-system states that it has rebooted it or turned it off. This means that the failure of a node implies the permanent failure of all services on it until/unless the node can be STONITH’d.

To solve this problem I have written a quick Xen STONITH module. The first issue is how to communicate between the DomU’s (Xen virtual machines) and the Dom0 (the physical host). It seemed that the best way to do this is to ssh to special accounts on the Dom0 and then use sudo to run a script that calls the Xen xm utility to actually restart the node. That way the Xen virtual machine gets limited access to the Dom0, and the shell script could even be written to allow each VM to only manage a sub-set of the VMs on the host (so you could have multiple virtual clusters on the one physical host and prevent them from messing with each other through accident or malice).

xen ALL=NOPASSWD:/usr/local/sbin/xen-stonith

Above is the relevant section from my /etc/sudoers file. It allows user xen to execute the script /usr/local/sbin/xen-stonith as root to do the work.

One thing to note is that from each of the DomU’s you must be able to ssh from root on the node to the specified account for the Xen STONITH service without using a password and without any unreasonable delay (IE put UseDNS no in /etc/ssh/sshd_config.

The below section (which isn’t in the feed) there are complete scripts for configuring this.

Continue reading →

Backup for Wind Power

June 18, 2007 | etbe

A question that people often ask about wind power (and was asked in the comments section of my previous post) is what can be done when the wind speed decreases in an area. There are several methods that can be used to address this problem.

The easiest option is to simply have wind farms spread out over a large area and interconnects that can spread the load. This greatly reduces the problems but is not a total solution.

The next step is to have a series of power plants that can quickly ramp up supply to meet the demand. One good option for this is gas-fired power plants, while they aren’t ideal for the environment they are cheap to build and can react quickly to changing demand. If a gas fired plant is only used when wind speeds are low it should on average be running at a small fraction of it’s peak capacity and use little fuel. Another good option is hydro-electric power which can be turned on quickly, which doesn’t produce any CO2 emissions and is already used widely (about 10% of Australia’s electricity is provided by hydro-electric power).

The ideal solution is to have every user of grid power know when the electricity is cheap (when there is a surplus of wind power) and when it’s expensive (when gas or hydro power is being used). Then non-critical services can be run when electricity is cheap. For example you could put clothes in your washing machine and program it to start the wash when electricity becomes cheap, some time during the day there will be a cheap time and the washing will get done. Once consumers know when electricity is cheap (via X10 or similar technology) they can use that information to determine when to use electricity generated from photo-voltaic cells on their roof and when to use grid power. The same technology can be used for heating and cooling of your home or office, turning off the A/C for an hour or so is only going to be a problem in the middle of summer or winter, for most of the year any heating or cooling could be done with cheap electricity. These technologies are all being developed at the moment (I once briefly worked on a system that could be used as a pre-cursor to managing home electricity use for times of cheap electricity).

Prius vs small non-hybrid car

June 11, 2007 | etbe

In response to my comment and post about the cost of driving to work there was a comment on the blog post I responded to suggesting that a small car is better value for money than a hybrid car.

The claim was made regarding a Nissan Pulsar, but to investigate this I decided to compare the Prius with the Corolla Hatchback, as far as I can tell the Corolla Hatchback is the nearest non-hybrid car to the Prius that Toyota sells (being similar in size, weight, and performance). Comparing cars of different make adds extra variables into the equation. Unfortunately the Toyota web site fails to provide specifications for the Prius and only provides a PDF file with minimal information on the Corolla, but it is enough for some minimal calculations.

A car company run by intelligent people would publish the specs on all their vehicles and provide a search form to compare selected models. The Standard Performance Evaluation Corporation has a good search method that allows easy comparison and ranking of items in their database (here is an example). It would be good if Toyota would permit us to compare models in their car range in a similar manner.

According to the Toyota web site the Prius uses 4.4L/100Km when driving in the city and the manual transmission version of the Corolla Hatchback uses 7.4L/100Km (7.7 for the auto). For the average 16,000Km that an average Prius owner drives that would save 480L of petrol which would save about $700 at current petrol prices.

The Corolla Ascent Hatchback (the cheapest of all Corollas) is $21,000 while the cheapest Prius is $37,400. If you compare the Prius with the cheapest Corolla then it’s a $16,400 price difference. If you save $700 per annum then it won’t cover the interest on a $16,400 loan or match the interest rates earned by a bank term-deposit if the $16,400 was invested. So it seems apparent that at current petrol prices (NB petrol prices are expected to increase) and with average Prius driving patterns a Prius will not be more economical than a cheap Corolla.

Currently the Commonwealth Bank of Australia offers 6.05% interest on term deposits of between $10,000 and $25,000. This means that $700 per annum would be the interest on a term deposit of $11,500. If we compare the Corolla Ultima Sedan at $32,000 with the Prius at $37,400 the difference in price is less than $11,500 – but the cars have incomparable sets of luxury and safety features. The Prius i-tech appears to have a super-set of the luxury and safety features of the Corolla Ultima Sedan but at $46,900 is again going to cost more for the average Prius driver.

The Prius is a very quiet car to drive, there is almost no engine noise (when driving at speeds where the Petrol engine is operating there is usually more noise from other vehicles) and no gear changes (handy if passengers are consuming hot or sticky drinks). It has a good set of safety and luxury features and is also a prestige car (no-one will say “oh wow, you’ve got a Corolla”). If you assign a dollar value to these features then a Prius may be the most economical car that meets your requirements!

Finally, let’s keep in mind the fact that petrol prices are steadily increasing. If you save $700 by driving a Prius this year then you may save $1000 next year. There is also the option of converting a Prius to a plug-in hybrid which will be a useful option if petrol prices hit $10/L! Also the amount of money saved will depend on the use of the car. If you are running a courier or taxi business then a Prius will probably be a lot more economical than a Corolla due to the greater distances travelled and the travel in the slow city traffic that the Prius was designed for.

PS All prices are in Australian dollars and concern products on offer in Australia, I would like to see comments from other people who perform the same calculations for their countries.

Update: If this interests you then you may want to read other posts I filed under the Environment and Cars categories.

Another Heartbeat 2.0 STONITH example configuration

May 30, 2007 | etbe

In a Heartbeat cluster installation it may not be possible to have one STONITH device be used to reboot all nodes. To support this it is possible to have multiple STONITH devices configured that will each be used to reboot different nodes in the cluster. In the following code section there is an example of how to configure STONITH for two separate ssh instances. Of course this is not useful apart from as an example of how to configure STONITH. It would be quite easy to change one of those ssh configuration entries to use IPMI or some more effective method of managing machines. My previous post on this topic has an example of a simpler ssh STONITH configuration.

It is convenient that the ssh method for rebooting nodes is available both as a shared object (which is used by the following example XML) and as a shell script (type external/ssh). The shell script can be used to give the same functionality as the shared object (with reduced performance) but the benefit is as an example of how to write external plugins. For a client I have just written an IPMI module that works with machines that have two Ethernet ports. When a server has two Ethernet ports you want to send an IPMI reset command to both of them in case the failure which requires a STONITH was triggered by a broken Ethernet cable. Unfortunately I can’t release the IPMI code at this time

Continue reading →

SE Linux in Debian

May 18, 2007 | etbe

I have now got a Debian Xen domU running the strict SE Linux policy that can boot in enforcing mode. I expect that tomorrow I will have it working with full functionality and that I will be able to run another SE Linux Play Machine in the near future.

After getting the strict policy working I want to build a Debian kernel with CONFIG_AUDITSYSCALL and an audit package so that I can audit system calls that an application makes and also so that the auditd can collect the SE Linux log messages. Other people have talked about packaging audit for Debian, hopefully one of them will do it first and save me the effort, but it shouldn’t be too difficult to do if they don’t.

Then I need to investigate some options for training people about SE Linux. As I don’t currently have the bandwidth for serving large files I’m thinking of basing some SE Linux training on Xen images from the jailtime.org repository. My rough plan at the moment is to have people download Xen images, run through them while consulting a web page, and ask questions on an IRC channel. I’m not sure what the demand will be for this but some web pages teaching people about SE Linux will be a useful resource even if the IRC based training doesn’t work out.

Another thing I want to do is to get PolyInstantiated Directories working in Debian. The pam_namespace.so module needed for this is written for a more recent version of PAM, so I might just work on merging the Debian patches with the latest upstream PAM instead of back-porting the module to the ancient Debian PAM.

permalinks in wordpress, Apache redirection, and other blog stuff

May 6, 2007 | etbe

When I first put my new blog online I didn’t think to set the custom permalinks option to avoid having /index.php in all URLs (which wastes a few bytes and looks nasty).

So I decided to change to better URLs but unfortunately many people have already bookmarked the bad URLs. I wanted to give a HTTP 301 redirection when someone uses the old index.php version (so that bookmarks get updated) and then redirect to the PHP file. Unfortunately having a redirection from ^/index.php to a version without it and then a local rewrite to include index.php again doesn’t seem to work (any advice would be appreciated). So I put the following in my /etc/wordpress/htaccess file (the location for such things in Debian) so that foo.php is used instead where foo.php is a sym-link to index.php. I’m wondering whether I should file a bug report against the Debian package requesting that a sym-link be in the package to facilitate such things – if it’s not possible to do what I desire without the symlink.

RewriteEngine On
RewriteBase /
#RewriteCond %{REQUEST_URI} ^/index.php/?(.*$) [NC]
#RewriteRule . /%1 [R=301,L]
RewriteCond ^/robots.txt [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
#RewriteRule . /foo.php%1 [L]
RewriteRule . /index.php%1 [L]

Update: I am now using the permalink-redirect plugin (thanks for the tip Method) which solves the problem of the obsolete URLs as well as solving the problem of having two representations of the URL (with and without a trailing slash). I have updated the above htaccess file sample to reflect my new configuration (with the old settings commented out for the benefit of people who don’t want the permalink-redirect plugin).

The way WordPress allows the table prefix to be stored in the MySQL configuration section is very handy. Some time ago I asked for advice on a blog server for multiple users and WordPress-MU was recommended, but it seems that for most situations where you want multiple blogs the non-MU version of WordPress will do the job. It seems that the main benefit of WordPress-MU is that setting up multiple blogs doesn’t require running shell scripts, which for the cases I’m most interested in doesn’t compete with the benefit that the non-MU version has of being packaged in Debian.

On the topic of WordPress in Debian, it’s a pity that none of the plugins are packaged in Debian. I plan to create a repository for plugins and themes that I use if no-one else has started such a repository. I believe that a repository of Debian packages for such things will provide significant benefits to users, including updates for security reasons and having plugins that are known to work (some of the plugins appear to only work on Windows).

Also there are a few issues that I would like to improve in WordPress. One is that the Uncategorised category is selected by default so if I select another category and forget to de-select Uncategorised then it’s a little confusing. Another is that the categories are displayed in the side-bar without mentioning the number of matching posts. The way blogger lists the number of posts per category (and sorts the categories in order) is much more convenient. Also another advantage of blogger is the handling of archives where you can click on a month to see a list of the names of all posts in that month. I’m not about to go back, but it would be nice to have those features. Does anyone have any ideas how to solve these problems?

Update2:
I have added a rule to make robots.txt not redirect. Before adding this rule /robots.txt was redirected to /index.php/robots.txt which caused a WordPress page to load, this wasted a lot of bandwidth (robots.txt is hit often) and probably caused some spiders to ignore my site.

school rating

May 4, 2007 | etbe

The web site http://au.ratemyteachers.com/ allows Australian students to rate their teachers. Ratings are anonymous and give teachers a score out of 5 as well as allowing students to comment on teachers.

The Sydney Morning Herald has an article about the site that describes the actions that the NSW Department of Education and the NSW Teachers Federation are taking to block the site.

The solution to this however is really quite simple. There needs to be a formal method for students to rate their teachers which will be used when it comes time to give pay rises to good teachers and dismiss or transfer to non-teaching duties the teachers who can’t do their job.

I encourage students to submit essays and debate topics about the anonymous news-papers published in the Soviet Union and other repressive states, why they were necessary (because criticism of the government was prohibited) and why they were morally right (a system with no method of correction will inevitably do bad things). Then teachers will have a choice of supporting the actions of the Soviet Union or the use of ratemyteacher.com, it will be interesting to see which option they choose. I think that it’s most likely that they will take the hypocritical path and support anonymous newspapers in the Soviet Union while attacking such free speech in supposedly free countries.

It’s interesting that an article on the failures of Mentone Grammar has just been published. Maybe if Mentone had been listed on the ratemyteachers.com site the Taylor’s would not have made the mistake of sending their son there. Or maybe if the Mentone senior staff had been reading that site they would have been able to correct the problems before they became cause for a legal dispute.

installing Xen domU on Debian Etch

January 1, 2007 | etbe

I have just been installing a Xen domU on Debian Etch. I’ll blog about installing dom0 later when I have a test system that I can re-install on (my production Xen machines have the dom0 set up already). The following documents a basic Xen domU (virtual machine) installation that has an IP address in the 10.0.0.0/8 private network address space and masquerades outbound network data. It is as general as possible.

lvcreate -n xen1 -L 2G /dev/vg

Firstly use the above command to create a block device for the domU, this can be a regular file but a LVM block device gives better performance. The above command is for a LV named xen1 on an LVM Volume Group named vg.

mke2fs -j /dev/vg/xen1

Then create the filesystem with the above command.

mount /dev/vg/xen1 /mnt/tmp
mount -o loop /tmp/debian-testing-i386-netinst.iso /mnt/cd
cd /mnt/tmp
debootstrap etch . file:///mnt/cd/
chroot . bin/bash
vi /etc/apt/sources.list /etc/hosts /etc/hostname
apt-get update
apt-get install libc6-xen linux-image-xen-686 openssh-server
apt-get dist-upgrade

Then perform the basic Debian install with the above commands. Make sure that you change to the correct directory before running the debootstrap command. The /etc/hosts and /etc/hostname files need to be edited to have the correct contents for the Xen image (the default is an empty /etc/hosts and /etc/hostname has the name of the parent machine). The file /etc/apt/sources.list needs to have the appropriate configuration for the version of Debian you use and for your preferred mirror. libc6-xen is needed to stop a large number of kernel warning messages on boot. It’s a little bit of work before you get the virtual machine working on the network so it’s best to do these commands (and other package installs) before the following steps. After the above type exit to leave the chroot and run umount /mnt/tmp.

lvcreate -n xen1-swap -L 128M /dev/vg
mkswap /dev/vg/xen1-swap

Create a swap device with the above commands.

auto xenbr0
iface xenbr0 inet static
pre-up brctl addbr xenbr0
post-down brctl delbr xenbr0
post-up iptables -t nat -F
post-up iptables -t nat -A POSTROUTING -o eth0 -s 10.1.0.0/24 -j MASQUERADE
address 10.1.0.1
netmask 255.255.255.0
bridge_fd 0
bridge_hello 0
bridge_stp off

Add the above to etc/network/interfaces and use the command ifup xenbr0 to enable it. Note that this masquerades all outbound data from the machine that has a source address in the 10.1.0.0/24 range.

net.ipv4.conf.default.forwarding=1

Put the above in /etc/sysctl.conf, run sysctl -p and echo 1 > /proc/sys/net/ipv4/conf/all/forwarding to enable it.

cp /boot/initrd.img-2.6.18-5-xen-686 /boot/xen-initrd-18-5.gz

Set up an initial initrd (actually initramfs) for the domU with a command such as the above. Once the Xen domU is working you can create the initrd from within it which gives a smaller image.

kernel = "/boot/vmlinuz-2.6.18-5-xen-686"
ramdisk = "/boot/xen-initrd-18-5.gz"
memory = 64
name = "xen1"
vif = [ "" ]
disk = [ "phy:/dev/vg/xen1,hda,w", "phy:/dev/vg/xen1-swap,hdb,w" ]
root = "/dev/hda ro"
extra = "2 selinux=1 enforcing=0"

The above is a sample Xen config file that can go in /etc/xen/xen1. Note that this will discover an appropriate bridge device by default, if you only plan to have one bridge then it’s quite safe, if you want multiple bridges then things will be a little more complex. Also note that there are two block devices created as /dev/hda and /dev/hdb, obviously if we wanted to have a dozen block devices then we would want to make them separate partitions with a virtual partition table. But in most cases a domU will be a simple install and won’t need more than two block devices.

xm create -c xen1

Now start the Xen domU with the above command. The -c option means to take the Xen console (use ^] to detach). After that you can login as root at the Xen console with no password, now is a good time to set the password.

Run the command apt-get install udev, this could not be done in the chroot before as it might mess up the dom0 environment. Edit /etc/inittab and disable gettys on tty2 to tty6, I don’t know if it’s possible to use them (the default and only option for xen console commands is tty1) and in any case you would not want 6, saving a few getty processes will save some memory.

Now you should have a basically functional Xen domU. Of course a pre-requisite for this is having a machine with a working dom0 installation. But the dom0 part is easier (and I will document it in a future blog post).

etbe – Russell Coker

Linux, politics, and other interesting things

Tag Archives: Most Popular