Bugs and User Practice

April 18, 2008 | etbe

Wouter points out a mistake in one of my blog posts which was based on old data [1]. My original post was accurate for older distributions of Linux but since then the bug in question was fixed.

Normally when writing blog posts or email I do a quick test before committing the text to avoid making mistakes (it’s easy to mis-remember things). However in this case the bug would dead-lock machines which made me hesitant to test it (I didn’t have a machine that I wanted to dead-lock).

There are two lessons to be learned from this. The most obvious is to test things thoroughly before writing about them (and have test machines available so that tests which cause service interruption or data loss can be performed).

The next lesson is that when implementing software you should try not to have limitations that will affect user habits in a bad way. In the case of LVM, if the tool lvetend had displayed a message such as “Resizing the root LV would dead-lock the system, until locking is fixed such requests will be rejected – please boot from a rescue disk to perform this operation” then I would have performed a test before writing the blog post (as it would be a harmless test to perform). Also on occasion when I really wanted to resize a root device without a reboot I would have attempted the command in the hope that LVM had been fixed.

A bug that deadlocks a system is one that will really have an adverse affect on users, both their habits in future use, and the probability of them using the software in future. A bug (or missing feature) that displays a warning message will have much less of a problem.

From now on I will still be hesitant in using lvextend on a LV for a root filesystem on any machines other than the very latest for fear that they will break. The fact that lvextend will sometimes work on the root filesystem and sometimes put the machine offline is a serious problem that impacts my use of that feature.

Most people won’t be in a position to have a bug or missing feature that deadlocks a system, but there are an infinite number of ways that software can potentially interrupt service or destroy data. Having software fail in a soft way such that data is not lost is a significant benefit for users and an incentive to use such software.

I’ve put this post in the WTF category, because having a dead-lock bug in a very obvious use-case of commonly used software really makes me say WTF.

[1] http://www.grep.be/blog/en/computer/cluebat/actually_you_can

Solar Powered PC

April 18, 2008 | etbe

I’ve just read an interesting post on TomsHardware.com about a solar powered PC [1]. It describes all the steps involved in creating a modern high-performance low-power computer.

They have a lot of interesting information. One surprising fact (from page 3) is that the PSUs tested (both for AC and DC input) were more efficient when idle (I expected the greatest efficiency to be when under load).

An AMD processor was chosen due in large part to the fact that chipsets in suitable motherboards used less power. For the CPU itself Intel had a competitive offering but no matching motherboard was power efficient enough (from page 7).

Page 8 documents how using a cooling fan (instead of passive cooling) reduced the power requirements of the CPU to such a degree that it always saved power use overall. Why do CPUs take less power when they are cooler?

Page 9 mentions that a small passively cooled video card can draw 88.5W when idle! That sucks pretty badly, it seems that having a video controller integrated with the motherboard is the way to go if you want to save power.

It’s interesting to note how much energy can be used by RAM. Page 13 shows that the difference between 2*1G and 2*512M can be as much as 3.4W and that the difference between different brands of RAM for the 2*1G can make as much as 1.2W difference. Their final system drew 61W when idle, my latest 64bit system takes 52W when idle [2] (which compares to the 38W of their system without a monitor), so we are talking about 9% of system power being saved by using less RAM or 3% being saved by using a different brand of RAM.

The summary of hard drive power use on page 14 is interesting, the fact that 2.5 inch laptop disks use less power than 3.5 inch desktop disks is hardly surprising, but the difference when idle is very surprising (apparently one of the 3.5 inch disks spends 8W on turbulence and friction in the bearings). It’s unfortunate that they didn’t compare any of the server-class 2.5 inch disks, it was about 6 months before the article was written that HP announced that in future they would cease shipping 3.5 inch disks and only use 2.5 inch disks (I wonder if this is related to all HP’s recent work on server cooling). Rumor has it that many server class 3.5 inch disks have platters that would fit into a 2.5 inch case because at high rotational speeds a larger diameter platter would not be strong enough.

The information on DVD power use on page 15 is quite shocking. From now on when I install machines as servers which don’t have a need for for a CD-ROM drive I’ll remove the drive prior to deployment. Even if it saves only 0.47W then it’s still worth doing on a machine which uses less than 40W! An additional benefit of this is that it might speed up the boot process as the system won’t need to check for a bootable CD.

It’s unfortunate that most computer parts don’t have published documentation on how much power they draw. Even if you don’t want to run on solar power there are still significant benefits to saving electricity (including reducing the noise from cooling fans and heat problems in summer). If technical data was published then people could make informed decisions about which parts to buy.

Update: Changed the percentage savings for different types of RAM to be based on the system power use without the monitor. I’m most interested in saving power for servers and for idle desktops (running a desktop machine 24*7 is pretty common) so most of the time the monitor will be turned off.

It’s interesting to note that they power their system uses is about the same as a P3 system and could be less if they used a different hard drive.

Making Linux DVDs

April 17, 2008 | etbe

Anthony Towns writes about using an improved version of jigdo to download CD/DVD images [1]. His improvement is basically to pipeline operation for better performance.

Jigdo (the Jigsaw download) is a tool to download a set of files and then use them to create a CD or DVD image [2]. The idea is that most web sites that have CD or DVD images also have a collection of files which comprise the DVD image available. This removes the need to store the data twice (wasting disk space on mirrors and in some situations breaking web caching).

I have never used jigdo, and for all Debian installations in recent times (the last few years at least) I download a small net-inst CD image and then have it download the other files from the net. I have Squid set up to cache large objects so this doesn’t waste too much of my precious network bandwidth (which is limited and expensive in Australia).

Now I’m thinking about what the optimum method for doing installs might be. One thing that would be good would be to support multiple repositories, the packages have unique file names and checksums so it should be possible to check one repository and then check another if it’s not working. I don’t mean multiple “deb” lines in the APT configuration. What I would like to do is to have an NFS file server or web server with an archive of downloaded packages and have APT check there first before downloading a file. So APT could get a list of packages from the net and then get the actual files locally if they are available.

The next thing that would be good is the ability to create a CD or DVD image dynamically and to store all temporary files. So I could download files from the repository and create a DVD image with just the packages that I need. Every time I create a DVD image my sub-set of the Debian archives would increase and the number of files actually downloaded in the creation process would be reduced. The effect would be to incrementally create a local mirror of the Debian repository.

Then I would like to see a two-stage DVD install process. I would like to boot from a CD or DVD and start the install and then have it give a list of files needed (which could be stored on a USB device or floppy) to create further CDs or DVDs for installation. One situation where this could have been beneficial was when I was doing an emergency install of CentOS. I did the first part of the install (selecting packages etc) to determine which CDs were needed. It turned out that almost all CDs were needed even though some of the CDs had only a few files that I was installing. If the installer could have written a list of packages to a USB device then I could have downloaded just those packages and got the install working a lot sooner. It seems to me that it’s fairly common to do one test install and then do some dozens of other installs with the same settings. So the ability to create a DVD of exactly the needed files for the other dozens of installs would be a great benefit.

Now this is just random commentary, unfortunately don’t have time to do any coding on this. But it seems obvious that something has to be done to improve the situation for developers and IT staff who need some degree of mirroring of the Debian package pool but who can’t do a full mirror. Back in 1996 I was able to mirror Debian over a 28K8 modem link and fit it on what was a reasonable hard drive by the standards of the day (incredibly tiny by today’s standards). Now I can’t practically mirror Debian over an Australian cable broadband connection and even by the standards of about 4 years ago (the age of most of my hard disks) the space requirements are significant.

I hope this post helps inspire some interest in developing these features. As delays in joining Debian [3] is the topic of the day it should be noted that work on preparing DVD images can easily be done by people who are not DDs. Such software should work from Debian archives without requiring any changes to them, and thus nothing special is needed from the Debian project to start work.

Motivation and Perspective

April 17, 2008 | etbe

Patrick Winnertz writes about the demotivating effect of unreasonable delays on joining the Debian project [1].

While I agree that things need to be improved in terms of getting people in the project in a timely manner (the suggestion of providing assistants seems good), I don’t think that anyone has a good reason for being demotivated because of this.

I first applied to join Debian in late 1998 or some time in 1999. At the time part of the process of joining was to receive a phone call. At the time I was living in a hotel and they refused to call me on such a line. I could have easily camped out in the hallway of a hotel (the cheap London hotels often had a pay-phone in the hall and no phones in the rooms) and pretended that it was my own phone with an unlisted number. Unless they refused to allow people with unlisted numbers to join (which seems unlikely) then I could have joined then. So it seems that at the time I could only join Debian if I was prepared to lie about my ownership of a phone line.

I wasn’t overly bothered by this – there has never been a shortage of free software projects that need contributions of code. By late 2000 the rules had changed and I joined without needing a phone call. In the mean time I had forked the Bonnie storage benchmark program to form my own project Bonnie++ [2], created Postal – mail server benchmark suite [3] and worked on many other things as well.

I have sympathy for the people who apply to become Debian Developers and who have to wait a long time, I’ve been in the same situation myself. But there are plenty of things that you can do in the mean time. Some of the things that you can do are upstream development work, filing bug reports, submiting patches that fix bugs, and writing documentation (all forms including blog posts). Also when projects aren’t yet in Debian it often happens that someone creates unofficial packages, the person who does this doesn’t need to be a DD. Producing back-ported packages for new versions of programs that are in a stable release can also be done by people who are not DDs. Unofficial and back-ported packages provide less benefit for the project as a whole but considerable benefit for the people who want to use them.

There is a lot of work that can be done to fulfill clause 4 of the Debian Social Contract [4] (Our priorities are our users and free software) which doesn’t require being a Debian developer. It seems to me that if you have the right approach to this and maintain the perspective that Debian is one part of the free software community (and not necessarily the biggest or most significant) then a delay in your application to become a DD won’t be particularly demotivating.

Resizing the Root Filesystem

April 16, 2008 | etbe

Uwe Hermann has described how to resize a root filesystem after booting from a live-cd or recovery disk [1]. He makes some good points about resizing an LVM PV (which I hadn’t even realised was possible).

The following paragraph is outdated, see the update at the end:
Incidentally it should be noted that if your root filesystem is an LVM logical volume then it can’t be resized without booting from a different device because the way LVM appears to work is that the LV in question is locked, then files under /etc/lvm/ are written, and then the LV is unlocked. If the LV in question contains /etc/lvm then you deadlock your root filesystem and need to press reset. Of course if your root filesystem is on an LV which has been encrypted via cryptsetup (the LV is encrypted not the PV) then a live resize of the root filesystem can work as locking the LV merely means that write-backs from the encryption layer don’t get committed. I’m not sure if this means that data written to an encrypted device is less stable (testing this is on my todo list).

If your root filesystem is on a partition of a hard drive (such as /dev/hda2) then it is possible to extend it without booting from different media. There is nothing stopping you from running fdisk and deleting the partition of your root filesystem and recreating it. When you exit fdisk it will call an ioctl() to re-read the partition table, the kernel code counts the number of open file handles related to the device and if the number is greater than 1 (fdisk has one open handle) then it refuses to re-read the table. So you can use fdisk to change the root partition and then reboot to have the change be noticed. After that ext2online can be used to take advantage of the extra space (if the filesystem has a recent enough version of the ext3 disk format).

One thing he didn’t mention is that if you do need to boot from another device to manipulate your root filesystem (which should be quite rare if you are bold and know the tricks) then you can always use your swap space. To do this you simply run swapoff and then run mkfs on the device that had been used for swap (incidentally there is nothing really special about the swap space, but it does tend to often be used for such recovery operations simply because it has no data that persists across a reboot).

The minimum set of files that need to be copied to a temporary filesystem is usually /bin, /sbin, /dev, /lib (excluding all directories under /lib/modules apart from the one related to the kernel you are using), and /etc. Also you should make directories /proc, /sys, and /selinux (if the machine in question runs SE Linux). The aim is not to copy enough files for the machine to run in a regular manner, merely enough to allow manipulating all filesystems and logical volumes. Often for such recovery I boot with init=/bin/bash as a kernel parameter to skip the regular init and just start with a shell. Note that when you use init=/bin/bash you end up with a shell that has no job control and ^C is not enabled, if you want to run a command that might not terminate of it’s own accord then the command “openvt /bin/bash” can be used to start another session with reasonable terminal settings.

I recommend that anyone who wants to work as a sys-admin experiment with such procedures on a test machine. There are lots of interesting things that you can learn and interesting ways that you can break your system when performing such operations.

Update: Wouter points out that the LVM bug of deadlocking the root filesystem has been fixed and that you can also use ext2online to resize the mounted filesystem [2].

Software Development is a Team Sport

April 16, 2008 | etbe

Albert writes about software development and how much teamwork is used [1]. He makes an interesting clash of analogies by suggesting that it’s not a “team sport” because “its not like commercial fishing where many hands are used to pull in the net at the same time“.

I think that software development for any non-trivial project is a team sport. You don’t have the same level of direct coordination as required for pulling in a net (or the rugby scrum [2] to use a sporting analogy), but that doesn’t stop it being a team effort.

Some parts of team development projects are like a relay event, in corporate environments the work is done in parallel simply because everyone is working the same hours but in free software development projects the work is often serialised. I think that it’s often more effective to serialise some work, if someone is actively working on one sections of code it may save time to avoid working in that area until they are finished. There is little benefit in writing code to old interfaces.

Some parts of team projects have specialised skill areas (EG debugging, skills in particular programming languages, and graphical design). Soccer is one sport where different rules apply to different players (the goal keeper can use their hands). In ice-hockey the protective clothing used by the goal keeper is considerably different from that used by other players. In most team sports where the aim is to put a ball through a goal at one end (EG basketball and all versions of football) there seems to be some degree of specialisation, some players are dedicated to scoring goals while others are dedicated to defense. The fielding team in cricket has every player assigned to a different part of the field – with slight differences in the skills required.

Then there is the issue of large projects such as Linux distributions. It seems to me that a Linux distribution will always comprise multiple teams as well as some individual projects. Maybe we could consider Linux distributions (and distributions of the other free OSs) to be similar to countries that compete in the Olympics. The culture of the Free Software (or Open Source if that’s your idea) community can be compared to the Olympic Spirit. Of course the Olympic idea that people should come together in peace for the Olympic Games and that it’s about honor not money is pretty much dead.

Maybe the Free Software development processes should be compared to an ideal of what sporting contests would be if there weren’t unreasonable amounts of money (and therefore corruption) involved.

Of course no analogy is perfect and there are many ways in which this one breaks down. One of which is the cooperation between distributions. There is a lot of private discussion between developers of various distributions and upstream developers about how to plan new features. It’s not uncommon for developers to announce certain development decisions as soon as they are made to help other distributions make decisions – for a developer in a distribution project if there is an issue which doesn’t matter much to you or your users then it’s often good to strive for compatibility with other distributions.

When users advocate new features or changes they sometimes try multiple distributions. It’s not uncommon for a feature request to be rejected by one distribution and then accepted by another. Once a feature is included in a major distribution the upstream developer is more likely to accept it due to it’s wide testing. Then when the feature is in upstream it’s almost certain to be included in all other distributions. I often recommend that when someone disagrees with one of their bugs being closed as “not a bug” that they try reproducing it in another distribution and reporting it there. As a side note, the criteria for reporting a bug in any free software distribution is that you can describe it in a way that allows other people to reproduce it – whether it’s a bug that afflicts you every day or whether you installed the distribution for the sole purpose of reporting the bug in a new forum is not relevant. As a general rule I recommend that you not have the same bug report open in more than one distribution at any time (if you notice a bug reported in multiple distributions then please add a note to each bug report so that work can be coordinated). As a general rule the only situation where I will open the same bug in multiple forums is if I have been told that the responsible person or people in one forum are unwilling or unable to fix it.

Finally, the people who consider that they don’t need to be a team player because they do their coding alone might want to consider Qmail. Dan Bernstein is a great coder and Qmail is by most metrics a fine piece of software, in terms of security Qmail is as good as it gets! If Dan was more of a team player then I believe that his mail server would have been much more successful (in terms of the number of sites using it). However I do understand his desire to have a great deal of control over his software.

Buying Old PCs

April 15, 2008 | etbe

I install quite a number of internet gateway machines for one of my clients. While eventually he will probably move to using an ASUS EeePC [1] or something similar, the current plan is to keep using desktop PCs (unfortunately server-class machines make too much noise).

P4 machines use an unreasonable amount of power and don’t seem worth getting second-hand. P3 machines are the most power efficient machines that have been commodity PCs (with keyboard and monitor), see my computer power use document for an example [2].

So what I would like is to get a dozen name-brand P3 systems at a reasonable price. Currently the only companies I can find selling such machines are charging ridiculous prices (such as $300) with the aim of reaming corporations who want to complete a set of old machines rather than upgrading them all. Most online auctions etc are selling P4 and Celeron 2GHz as the minimum hardware. The Celeron machines are better than P4 (at around 50W) but not as good as P3 (at less than 40W).

If you read this post and are in Melbourne, Australia and have a name-brand P3 desktop system that you want to sell then let me know. If you have any suggestions of how to buy such machines then I’m interested in advice.

Fried Food

April 15, 2008 | etbe

I’ve just been reading about oil deterioration when deep frying [1]. Apparently frying food can cause the production of Acrylamide [2] which causes cancer, is a neurotoxin, and does other bad things to your body. Deep frying in a vacuum [3] is a solution to this problem as the lower temperatures reduce the formation of the bad chemicals (and also increase the life of the oil thus making for cheaper fried products). Of course this means that deep frying in a pressure cooker as KFC does probably gives a result that is a lot worse for your health [4].

For some time I’ve been avoiding KFC due to concerns about the treatment of the chickens. I have been regularly eating fried chicken from a local store which looks and tastes very similar to KFC so presumably is cooked in the same way. I guess I’ll have to stop eating that now.

I wonder if toast is also bad in this regard. I expect that sausages would be a problem when cooked on a BBQ.

In other health news I’ve been reducing my meat consumption and I’m considering trying to entirely avoid farm raised animals to restrict my meat consumption to game meat, this should be good for my health and is also a lot better for the animals and the environment.

Friends and Adverts

April 14, 2008 | etbe

For some time I have been running Google Adsense adverts on my blog. Not long after I started running the adverts the revenue exceeded the amount of money I pay for net access, so it’s worth having.

I recently increased the amount of advertising by including an advert in the middle of the post and received one negative comment from a reader who doesn’t like reading content with adverts in the middle. I have just changed my advertising to have that advert spot in question be before the start of the content rather than in the middle (I’m not sure if that will make them more or less happy). It seems expected nowadays that there will be a moderate amount of advertising on all web sites which have good content and don’t charge membership fees, many of my blog posts cite as references pages on web servers run by media companies that have significant amounts of advertising.

Chris Samuel is considering implementing advertising [1] and is planning to not display adverts to “friends” (which means people who have commented or who are referred by social media sites).

One significant problem with this idea is that sometimes the adverts become part of the blog conversation. Google adverts give an idea of which of the things that people are prepared to pay money for are considered by the Google algorithm to be related to the post in question. In my experience it’s not uncommon for people who comment on blog posts (both through the blog interface and through private correspondence) to reference the contents of adverts. Also while writing this post I updated my post about car sharing because I discovered a third car-share company through the adverts on my own blog [2]. I know one professional journalist who blocks the adverts on his own site so that his writing will not be biased (or be perceived to be biased) by his advertisers (which makes sense given his situation), but for my situation it clearly makes sense to review the advertised offerings of all three companies that compete in this space in Melbourne and there is no potential for bias (incidentally all three of them are advertising on my blog post).

A problem with giving a different version of the content for friends (in the more traditional meaning of the word – as opposed to using the word to mean people who have visited your blog in a certain way) is that they will not be able to correctly review your web site. I regularly notify friends and people who do worthy things of deficiencies that I discover in their web sites and occasionally receive such notifications from other people. I don’t want to have bugs in my site concealed from friendly people who would like to help me out and displayed to random people who don’t care and will respond by visiting another site.

When such things are discussed one topic that is mentioned is putting advertising in an RSS feed. I have no plans to do this, such RSS advertising would not be acceptable to Planet installations, but it would not be technically challenging to give an advert free feed to Planets but give adverts to everyone else. I think that most people who read my blog through RSS feeds will see adverts when they want to comment on my posts or when they want to forward the URLs or reference them in their own blog posts (always click on the permalink before forwarding – when a blogger messes up their permalinks it’s embarrassing if you forward them without checking).

So I encourage Chris to put adverts on his blog and to show them to me.

Debian Work and Upstream

April 14, 2008 | etbe

Steve Kemp writes about security issues with C programs [1]. It seems obvious that if you are going to do something that is overly tricky (such as anything related to setuid programs) then you should have a good knowledge of what you are doing. Steve goes a little further and suggests that anyone who doesn’t know C should not package a C program.

Andrew Pollock is a Debian developer who doesn’t program in C and yet packages some C programs, he explains why he thinks that it’s correct to forward C coding bugs upstream [2].

In this debate I have more sympathy for Steve’s position. Security is one of the most important things in software that we develop, if you can’t develop secure software then IMHO you shouldn’t distribute any software. Although it should be noted that the two posts in question don’t directly conflict. Steve’s main point (your should know the code well to do something relevant to security) does not directly conflict with Andrew’s main point that a non-C coder can maintain the Debian packaging files and forward bug reports related to C code upstream.

In the more general case it’s often impossible to know all languages used in a project. It’s not uncommon for a single project to have core code written in C, a configuration file format (which is a programming language), a Makefile (GNU Make is a fairly complex interpreter), Autoconf/Automake (even more complex tools for creating complex Makefiles, and then some Perl or Python to manipulate input or output data. The SE Linux project has code in all those languages plus the M4 macro language. It seems obvious that in a large project the number of people who are capable of understanding all languages which are used is going to be a small sub-set of all Debian Developers. It also seems likely that someone who knows some of the languages really well may be capable of doing a better job than someone who has a passable knowledge of all of them.

For the vast majority of my work in Debian I have been a member of the upstream project in question (for Bonnie++, Postal, Portslave, Maildir-bulletin, and logtools I’m the sole upstream developer). Often I start maintaining a package with the intent of doing the minimum needed to keep it maintained in Debian but then end up doing upstream work, for example I briefly had write access to the CVS repository for pppd when I was maintaining it and I had write access to the KDE CVS repository when I was working on KDE packages (which incidentally was before I became an official DD).

When I first started working on the SE Linux project I aimed to merely package the code in question for Debian. I was not planning on spending seven years doing upstream work!

When I’m working on a project I spend my time working on the things that most need attention. That often means that I get diverted from Debian packaging bugs (which are often minor issues) to work on upstream bugs. In many ways my work practices make me a Debian user and upstream developer who also does Debian packaging rather than a dedicated DD. This of course does have a detrimental affect on the quality of my Debian packaging work – but overall I believe that it’s best for the users!

As an upstream developer who is focussed on Debian there are significant advantages in being a DD. I can immediately upload a new package to Unstable without any delay, and there is the potential to take over a related package (either be given the package or NMU) if it’s the best way to get around a roadblock. Another benefit is that in certain limited situations I can speak on behalf of Debian (in terms of packages I “own”). When it comes to upstream development of SE Linux I can make decisions about how things will be done in Debian and work with other upstream developers to ensure that cross-distribution compatibility works reasonably well.

It seems to me that a fairly ideal process for managing Debian development would have the people who are capable of the hard coding doing so and some people who haven’t got specific coding skills working on the packaging. I would be very interested in receiving patches related to the Debian packaging of the packages I “own” (in Debian it’s widely regarded that a man’s package is his castle) so that I can spend more of my time on the C coding. I’m not bothered about NMUs (a Non-Maintainer Upload – is when someone else uploads a new version of one of my packages to fix a bug). If anyone would like to NMU one of my packages they are welcome to do so as long as there is a suitable combination of bug urgency, simplicity of the fix, and lack of response from me. If someone wants to start NMUing my packages to fix bugs reported by the latest version of Lintian in Unstable (Lintian is the Debian tool to automatically check for packaging bugs) then that’s OK.

Of course there are a variety of other situations that DDs sometimes find themselves in. There have been situations of overtly hostile upstream developers. I try and avoid associating with people who have certain difficulties in interacting with others. If someone is going to change their license and try to back-date the change or create a license that makes it difficult to distribute modified versions of the code then I’m inclined to avoid even using the software, let alone being involved in developing it. This means that the software that I use in any serious way has reasonable people doing upstream development who welcome new people to join their work.

etbe – Russell Coker

Linux, politics, and other interesting things

Author Archives: etbe