Security Flaws in Free Software

May 21, 2008 | etbe

I just wrote about the system administration issues related to the recent Debian SSL/SSH security flaw [1]. The next thing we need to consider is how we can change things to reduce the incidence of such problems.

The problem we just had was due to the most important part of the entropy supply for the random number generator not being used due to a mistake in commenting out some code. The only entropy that was used was the PID number of the process which uses the SSL library code which gives us 15 bits of entropy. It seems to me that if we had zero bits of entropy the problem would have been discovered a lot sooner (almost certainly before the code was released in a stable version). Therefore it seems that using a second-rate source of entropy (which was never required) masked the problem that the primary source of entropy was not working. Would it make sense to have a practice of not using such second-rate sources of entropy to reduce the risk of such problems being undetected for any length of time? Is this a general issue or just a corner case?

Joss makes some suggestions for process improvements [2]. He suggests that having a single .diff.gz file (the traditional method for maintaining Debian packages) that directly contains all patches can obscure some patches. The other extreme is when you have a patch management system with several dozen small patches and the reader has to try and discover what each of them does. For an example of this see the 43 patches which are included in the Debian PAM package for Etch, also note that the PAM system is comprised of many separate shared objects (modules), this means that the patching system lends itself to having one patch per module and thus 43 patches for PAM isn’t as difficult to manage as 43 patches for a complex package which is not comprised of multiple separate components might be. That said I think that there is some potential for separating out patches. Having a distinction between different types of patches might help. For example we could have a patch for Makefiles etc (including autoconf etc), a patch for adding features, and a patch for fixing bugs. Then people reviewing the source for potential bugs could pay a lot of attention to bug fixes, a moderate amount of attention to new features, and casually skim the Makefile stuff.

The problem began with this mailing list discussion [3]. Kurt’s first message starts with “When debbuging applications” and ends with “What do you people think about removing those 2 lines of code?“. The reply he received from Ulf (a member of the OpenSSL development team) is “If it helps with debugging, I’m in favor of removing them“. It seems to me that there might have been a miscommunication there, Ulf may have believed that the discussion only concerned a debugging built and not a build that would eventually end up on millions of machines.

It seems possible that the reaction would have been different if Kurt had mentioned that he wanted to have a single source tree for both debugging and for regular use. It also seems likely that his proposed change may have received more inspection if he had clearly stated that he was doing to include it in Debian where it would be used by millions of people. When I am doing Debian development I generally don’t mention all the time “this code will be used by millions of people so it’s important that we get it right“, although I do sometimes make such statements if I feel that my questions are not getting the amount of consideration from upstream that a binary package destined for use by millions of people deserves. Maybe it would be a good practice to clarify such things in the case of important packages. For a package that is relevant to the security of the entire distribution (and possibly to other machines around the net – as demonstrated in this case) it doesn’t seem unreasonable to include a post-script mentioning the scope of the code use (it could be done with an alternate SIG if using a MUA that allows selecting from multiple SIGs in a convenient manner).

In the response from the OpenSSL upstream [4] it is claimed that the mailing list used was not the correct one. Branden points out that the openssl-team mailing list address seems not to be documented anywhere [5]. One thing to be learned from this is that distribution developers need to be proactive in making contact with upstream developers. You might think that building packages for a major distribution and asking questions about it on the mailing list would result in someone from the team noticing and mentioning any other things that you might need to do. But maybe it would make sense to send private mail to one of the core developers, introduce yourself, and ask for advice on the best way to manage communication to avoid this type of confusion.

I think that it is ideal for distribution developers to have the phone numbers of some of the upstream developers. If the upstream work is sponsored by the employer of one of the upstream developers then it seems reasonable to ask for their office phone number. Sometimes it’s easier to sort things out by phone than by email.

Gunnar Wolf describes how the way this bug was discovered and handled shows that the Debian processes work [6]. A similar bug in proprietary software would probably not be discovered nearly as quickly and would almost certainly not be fixed in such a responsible manner.

Update: According to the OpenSSL project about page [7], Ulf is actually not a “core” member, just a team member. I had used the term “core” in a slang manner based on the fact that Ulf has an official @openssl.org email address.

Debian SSH Problems

May 18, 2008 | etbe

It has recently been announced that Debian had a serious bug in the OpenSSL code [1], the most visible affect of this is compromising SSH keys – but it can also affect VPN and HTTPS keys. Erich Schubert was one of the first people to point out the true horror of the problem, only 2^15 different keys can be created [2]. It should not be difficult for an attacker to generate 2^15 host keys to try all combinations for decrypting a login session. It should also be possible to make up to 2^15 attempts to login to a session remotely if an attacker believes that an authorized key was being used – that would take less than an hour at a rate of 10 attempts per second (which is possible with modern net connections) and could be done in a day if the server was connected to the net by a modem.

John Goerzen has some insightful thoughts about the issue [3]. I recommend reading his post. One point he makes is that the person who made the mistake in question should not be lynched. One thing I think we should keep in mind is the fact that people tend to be more careful after they have made mistakes, I expect that anyone who makes a mistake in such a public way which impacts so many people will be very careful for a long time…

Steinar H. Gunderson analyses the maths in relation to DSA keys, it seems that if a DSA key is ever used with a bad RNG then it can be cracked by someone who sniffs the network [4]. It seems that it is safest to just not use DSA to avoid this risk. Another issue is that if a client supports multiple host keys (ssh version 2 can use three different key types, one for the ssh1 protocol, one for ssh2 with RSA, and one for ssh2 with DSA) then a man in the middle attack can be implemented by forcing a client to use a different key type – see Stealth’s article in Phrack for the details [5]. So it seems that we should remove support for anything other than SSHv2 with RSA keys.

To remove such support from the ssh server edit /etc/ssh/sshd_config and make sure it has a line with “Protocol 2“, and that the only HostKey line references an RSA key. To remove it from the ssh client (the important thing) edit /etc/ssh/ssh_config and make sure that it has something like the following:

Host *
Protocol 2
HostKeyAlgorithms ssh-rsa
ForwardX11 no
ForwardX11Trusted no

You can override this for different machines. So if you have a machine that uses DSA only then it would be easy to add a section:

Host strange-machine
Protocol 2
HostKeyAlgorithms ssh-dsa

So making the default configuration of the ssh client on all machines you manage has the potential to dramatically reduce the incidence of MITM attacks from the less knowledgable users.

When skilled users who do not have root access need to change things they can always edit the file ~/.ssh/config (which has the same syntax as /etc/ssh/ssh_config) or they can use command-line options to override it. The command ssh -o “HostKeyAlgorithms ssh-dsa” user@server will force the use of DSA encryption even if the configuration file requests RSA.

Enrico Zini describes how to use ssh-keygen to get the fingerprint of the host key [6]. One thing I have learned from comments on this post is how to get a fingerprint from a known hosts file. A common situation is that machine A has a known hosts file with an entry for machine B. I want to get the right key in machine C and there is no way of directly communicating between machine A and machine C (EG they are in different locations with no network access). In that situation the command “ssh-keygen -l -f ~/.ssh/known_hosts” can be used to display all the fingerprints of hosts that you have connected to in the past, then it’s a simple matter of grepping the output.

Docunext has an interesting post about ways of mitigating such problems [7]. One thing that they suggest is using fail2ban to block IP addresses that appear to be trying to do brute-force attacks. It’s unfortunate that the version of fail2ban in Debian uses /tmp/fail2ban.sock for it’s Unix domain socket for talking to the server (the version in Unstable uses /var/run/fail2ban/fail2ban.sock). They also mention patching network drivers to add entropy to the kernel random number generator. One thing that seems interesting is the package randomsound (currently in Debian/Unstable) which takes ALSA sound input as a source of entropy, note that you don’t need to have any sound input device connected.

When considering fail2ban and similar things, it’s probably best to start by restricting the number of machines which can connect to your SSH server. Firstly if you put it on a non-default port then it’ll take some brute-force to find it. This will waste some of the attacker’s time and also make the less persistent attackers go elsewhere. One thing that I am considering is having a few unused ports configured such that any IP address which connects to them gets added to my NetFilter configuration – if you connect to such ports then you can’t connect to any other ports for a week (or until the list becomes too full). So if for example I had port N configured in such a manner and port N+100 used for ssh listening then it’s likely that someone who port-scans my server would be blocked before they even discovered the SSH server. Does anyone know of free software to do this?

The next thing to consider is which IP addresses may connect. If you were to allow all the IP addresses from all the major ISPs in your country to connect to your server then it would still be a small fraction of the IP address space. Sure attackers could use machines that they already cracked in your country to launch their attacks, but they would have to guess that you had such a defense in place, and even so it would be an inconvenience for them. You don’t necessarily need to have a perfect defense, you only need to make the effort to reward ratio be worse for attacking you than for attacking someone else. Note that I am not advocating taking a minimalist approach to security, merely noting that even a small increment in the strength of your defenses can make a significant difference to the risk you face.

Update: based on comments I’m now considering knockd to open ports on demand. The upstream site for knockd is here [8], and some documentation on setting it up in Debian is here [9]. The concept of knockd is that you make connections to a series of ports which act as a password for changing the firewall rules. An attacker who doesn’t know those port numbers won’t be able to connect. Of course anyone who can sniff your network will discover the ports soon enough, but I guess you can always login and change the port numbers once knockd has let you in.

Also thanks to Helmut for advice on ssh-keygen.

Ideas to Copy from Red Hat

May 16, 2008 | etbe

I believe that the Red Hat process which has Fedora for home users (with a rapid release cycle and new versions of software but support for only about one year) and Enterprise Linux (with a ~18 month release cycle, seven years of support, and not always having the latest versions) gives significant benefits for the users.

The longer freeze times of Enterprise Linux (AKA RHEL) mean that it often has older versions of software than a Fedora release occurring at about the same time. In practice the only time I ever notice users complaining about this is in terms of OpenOffice (which is always being updated for compatability with the latest MS changes). As an aside, a version of RHEL or CentOS with a back-port of the latest OpenOffice would probably get a lot of interest.

RHEL also has a significantly smaller package set than Fedora, there is a lot of software out there that you wouldn’t want to support for seven years, a lot of software that you might want to support if you had more resources, and plenty of software that is not really of interest to enterprise customers (EG games).

Now there are some down-sides to the Red Hat plan. The way that they run Fedora is to have new releases of software instead of back-porting fixes. This means that bugs can be fixed with less effort (simply compiling a new version is a lot less effort than back-porting a fix), and that newer versions of the upstream code get tested. With some things this isn’t a problem, but in the past I have had problems with the Fedora kernel. One example was when I upgraded the kernel on a bunch of remote Fedora machines only to find that the new kernel didn’t support the network card, so I had to talk the users through selecting the older kernel at the GRUB menu (this caused pain and down-time). A problem with RHEL (which I see regularly on the CentOS machines I run) is that it doesn’t have the community support that Fedora does, and therefore finding binary packages for RHEL can be difficult – and often the packages are outdated.

I believe that in Debian we could provide benefits for some of our users by copying some ideas from Red Hat. There is currently some work in progress on releasing packages that are half-way between Etch and Lenny (Etch is the current release, Lenny will be the next one). The term Etch and a half refers to the work to make Etch run on newer hardware [1]. It’s a good project, but I don’t think that it goes far enough. It certainly won’t fulfill the requirements of people who want something like Fedora.

I think that if we had half-way releases of Debian (essentially taking a snap-shot of Testing and then fixing the worst of the bugs) then we could accommodate user demand for newer versions (making available a release which is on average half as old). Users who want really solid systems would run the full releases (which have more testing pre-release and more attention paid to bug fixes), but users who need the new features could run a half-way release. Currently there are people working on providing security support for Testing so that people who need the more recent versions of software can use Testing, I believe that making a half-way release would provide better benefits to most users while also possibly taking less resources from the developers. This would not preclude the current “Etch and a half” work of back-porting drivers, in the Red Hat model such driver back-ports are done in the first few years of RHEL support. If we were to really follow Red Hat in this regard the “Etch and a half” work would operate in tandem with similar work for Sarge (version 3.1 of Debian which was released in 2005)!

In summary, the Red Hat approach is to have Fedora releases aimed at every 6 months, but in practice coming out every 9 months or so and to have Enterprise Linux releases aimed at every year, but in practice coming out every 18 months. This means among other things that there can be some uncertainty as to the release order of future Fedora and RHEL releases.

I believe that a good option for Debian would be to have alternate “Enterprise” (for want of a better word) and half-way releases (comparable to RHEL and Fedora). The Enterprise releases could be frozen in coordination with Red Hat, Ubuntu, and other distributions (Mark Shuttleworth now refers to this as being a “pulse” in the free software community [], while the half-way releases would come out either when it’s about half-way between releases, or when there is a significant set of updates that would encourage users to switch.

One of the many benefits to having synchronised releases is that if the work in back-porting support for new hardware lagged in Debian then users would have a reasonable chance of taking the code from CentOS. If nothing else I think that making kernels from other distributions available for easy install is a good thing. There is a wide combination of kernel patches that may be selected by distribution maintainers, and sometimes choices have to be made between mutually exclusive options. If the Debian kernel doesn’t work best for a user then it would be good to provide them with a kernel compiled from the RHEL kernel source package and possibly other kernels.

Mark also makes the interesting suggestion of having different waves of code freeze, the first for the kernel, GCC, and glibc, and possibly server programs such as Apache. The second for major applications and desktop environments. The third for distributions. One implication of this is that not all distributions will follow the second wave. If a distribution follows the kernel, GCC, and glibc wave but not the applications wave it will still save some significant amounts of effort for the users. It will mean that the distributions in question will all have the same hardware support and kernel features, and that they will be able to run each others’ applications (except when the applications in question use system libraries from later waves). Also let’s not forget the possibility of running a kernel from distribution A on distribution B, it’s something I’ve done on many occasions, but it does rely on the kernels in question being reasonably similar in terms of features.

Release Dates for Debian

May 13, 2008 | etbe

Mark Shuttleworth has written an interesting post about Ubuntu release dates [1]. He claims that free software distributions are better able to meet release dates than proprietary OSs because they are not doing upstream development. The evidence that free software distributions generally do a reasonable job of meeting release dates (and Ubuntu does an excellent job) is clear.

But the really interesting part of his post is where he offers to have Ubuntu collaborate with other distributions on release dates. He states that if two out of Red Hat (presumably Enterprise Linux), Novell (presumably SLES), and Debian will commit to the same release date (within one month) and (possibly more importantly) to having the same versions of major components then he will make Ubuntu do the same.

This is a very significant statement. From my experience working in the Debian project and when employed by Red Hat I know that decisions about which versions of major components to include are not taken lightly, and therefore if the plan is to include a new release of a major software project and that project misses a release date then it forces a difficult decision about whether to use an older version or delay the release. For Ubuntu to not merely collaborate with other distributions but to instead follow the consensus of two different distributions would be a massive compromise. But I agree with Mark that the benefits to the users are clear.

I believe that the Debian project should align it’s release cycles with Red Hat Enterprise Linux. I believe that RHEL is being released in a very sensible manner and that the differences of opinion between Debian and Red Hat people about how to manage such things are small. Note that it would not be impossible to have some variations of version numbers of components but still stick mostly to the same versions.

If Debian, Ubuntu, and RHEL released at about the same time with the same versions of the kernel, GCC, and major applications and libraries then it would make it much easier for users who want to port software between distributions and run multiple distributions on the same network or the same hardware.

The Debian Social Contract [2] states that “Our priorities are our users and free software“. I believe that by using common versions across distributions we would help end-users in configuring software and maintaining networks of Linux systems running different distributions, and also help free software developers by reducing the difficulty in debugging problems.

It seems to me that the best way of achieving the goal that Mark advocates (in the short term at least) is for Debian to follow Red Hat’s release cycle. I think that after getting one release with common versions out there we could then discuss how to organise cooperation between distributions.

I also believe that a longer support cycle would be a good thing for Debian. I’m prepared to do the necessary work for the packages that I maintain and would also be prepared to do some of the work in other areas that is needed (EG back-porting security fixes).

The Future of Xen

May 2, 2008 | etbe

I’m currently in Xen hell. My Thinkpad (which I won’t replace any time soon) has a Pentium-M CPU without PAE support. I think that Debian might re-introduce Xen support for CPUs without PAE in Lenny, but at the moment I have the choice of running without Xen or running an ancient kernel on my laptop. Due to this I’ve removed Xen from my laptop (I’m doing most of my development which needs Xen on servers anyway).

Now I’ve just replaced my main home server. It was a Pentium-D 2.8GHz machine with 1.5G of RAM and a couple of 300G SATA disks in a RAID-1. Now it’s a Pentium E2160 1.8Ghz machine with 3G of RAM with the same disks. Incidentally Intel suck badly, they are producing CPUs with names that have no meaning, and most of their chipsets don’t support more than 4G of physical address space [1]. I wanted 4G of RAM but the machine I was offered only supported addressing 4G and 700M of that was used for PCI devices. For computation tasks it’s about the same speed as the old Pentium-D, but it has faster RAM access, more RAM, uses less power, and makes less noise. If I was going to a shop to buy something I probably would have chosen something different to get support for more than 4G of RAM, but as I got the replacement machine for free as a favor I’m not complaining!

I expected that I could just install the new server and have things just work. There were some minor issues such as configuring X for the different video hardware (and installing the 915resolution package (which is only needed in Etch) to get the desired 1650×1400 resolution. But for the core server tasks I expected that I could just move the hard drives across and have it work.

After the initial install the system crashed whenever I did any serious hard drive access from Dom0, the Dom0 kernel Oopsed and network access was cut off from the DomU’s (I’m not sure whether the DomU’s died but without any way of accessing them it doesn’t really matter much). As a test I installed the version of the Xen hypervisor from Unstable and it worked. But the Xen hypervisor from Unstable required the Xen tools from Unstable which also required the latest libc6, and therefore the entire Dom0 had to be upgraded. Then in an unfortunate accident unrelated to Xen (cryptsetup in Debian/Unstable warns you if you try to use a non-LUKS option on a device which has been used for LUKS and would have saved me) I lost the root filesystem before I finished the upgrade.

So I did a fresh install of Debian/Unstable, this time it didn’t crash on heavy disk IO, instead it would lock up randomly when under no load.

I’ve now booted a non-Xen kernel and it’s working well. But this situation is not acceptable long-term, a large part of the purpose of the machine is to run virtualisation so that I can test various programs under multiple distributions. I think that I will have to try some other virtualisation technologies. The idea of running KVM on real servers (ones that serve data to the Internet) doesn’t thrill me, Tavis Ormandy’s paper about potential ways of exploiting virtual machine technologies [2] is a compelling argument for para-virtualisation. Fortunately however my old Pentium-3 machines running Xen seem quite reliable (replacing both software and hardware is a lot of pain that I don’t want).

In the near future I will rename the Xen category on my blog to Virtualisation. For older machines Xen is still working reasonably well, but for all new machines I expect that I will have to use something else – and I’ll be blogging about the new machines not the old. I expect that an increasing number of people will be moving away from Xen in the near future. It doesn’t seem to have the potential to give systems that are reliable when running on common hardware.

Ulrich Drepper doesn’t have a high opinion of Xen [3], the more I learn about it the more I agree with Ulrich.

Offensive Blog Posts

April 24, 2008 | etbe

There has been ongoing debate in the Debian community for a number of years about what standards of behavior should be expected. Matthew Garrett sets a new low by making a joke about Jesus being molested as a child [1]. While I believe that debate and discussion about religion is a good thing, such comments about someone who is widely regarded as a God (part of the Holy Trinity) seems to provide no value and just needlessly offends people. I used to be a Christian, and while I have great disagreements with most Christians about issues of religion I still believe that Jesus (as described in the bible) was a good person and deserves some respect. I don’t believe that blasphemy should be illegal, but some minimum standards should be observed when discussing religion.

Next there is the issue of child molesting, most people agree that there’s nothing amusing in that – so I hope that nothing more needs to be said about violating babies.

Finally there is the issue of rape in general often being treated as a joke in the computer industry (I am not sure how prevalent this is in the wider community). One example that is from a Wired article: “We were raped by Microsoft. Bill Gates did it personally. Introducing Gates to the president of a small company is like introducing Mike Tyson to a virgin” [2]. I admit that finding examples of this on the web is not easy, part of it is due to such slang use being more common in spoken communication than in written communication, another is the vast number of slang terms that are used.

A Google search for “male rape” [3] turns up some informative articles. One FAQ suggests that 3% of men will be raped as an adult [4] – I expect that some guys will decide that it’s not so funny when they realise that it could happen to them.

For people who’s knowledge of the English language is not as good as that of Matthew and I, here is the dictionary definition of the word “violated” [5].

The Purpose of Planet Debian

April 21, 2008 | etbe

An issue that causes ongoing discussion is what is the purpose of a Planet installation such as Planet Debian [1]. The discussion usually seems to take the less effective form of what is “appropriate” content for the Planet or what is considered to be “abuse” of the Planet. Of course it’s impossible to get anything other than a rough idea of what is appropriate is the purpose is not defined, and abuse can only be measured on the most basic technical criteria.

My personal use of Planet Debian and Planet Linux Australia [2] is to learn technical things related to Linux (how to use new programs, tricks and techniques, etc), to learn news related to Linux, and to read personal news about friends and colleagues. I think that most people have some desire to read posts of a similar nature (I have received a complaint that my blog has too many technical posts and not enough personal posts), but some people want to have a Planet with only technical articles.

In a quick search of some planets the nearest I found to a stated purpose of a Planet installation was from the Wiki to document Planet Ubuntu [3] which says ‘Subscribed feeds ought to be at least occasionally relevant to Ubuntu, although the only hard and fast rule is “don’t annoy people”‘. Planet Perl [4] has an interesting approach, they claim to filter on Perl related keywords, I initially interpreted this to mean that if you are on their list of blogs and you write a post which seems to refer to Perl then it will appear – but a quick browse of the Planet shows some posts which don’t appear to match any Perl keywords. Gentoo has implemented a reasonable system, they have a Universe [5] configuration which has all blog posts by all Gentoo bloggers as well as a Planet installation which only has Gentoo related posts.

It seems to me that the a reasonable purpose for Planet Debian would be to have blog feeds which are occasionally specific to Debian and often relevant to Debian. Personal blog posts would be encouraged (but not required). Posts which are incomprehensible or have nothing to say (EG posts which link to another post for the sole purpose of agreeing or disagreeing) would be strongly discouraged and it would be encouraged to make links-posts rare.

Having two installations of the Planet software, one for posts which are specific to Debian (or maybe to Debian or Linux) and one for all posts by people who are involved with Debian would be the best option. Then people who only want to read the technical posts can do so, but other people can read the full list. Most blog servers support feeds based on tag or category (my blog already provides a feed of Debian-specific posts). If we were going to have a separate Planet installation for only technical posts then I expect that many bloggers would have to create a new tag for such posts (for example my posts related to Debian are in the categories Benchmark, Linux, MTA, Security, Unix-tips, and Xen) and the tag Debian is applied to only a small portion of such posts. But it would be easy to create a new tag for technical posts.

Ubuntu is also the only organisation I’ve found to specify conditions upon which blogs might be removed from the feed, they say: We reserve the right to remove any feed that is inaccessible, flooding the page, or otherwise interfering with the operation of the Planet. We also have the right to move clearly offensive content or content that could trigger legal action.

That is reasonable, although it would be good to have a definition for “flooding the page” (I suggest “having an average of more than two posts per day appear over the period of a week or having posts reappear due to changing timestamps”). Also the “could trigger legal action” part is a minor concern – product reviews are often really useful content on a Planet…

Some time ago my blog was removed from Planet Fedora for some reason. I was disappointed that the person who made that change didn’t have the courtesy to inform me of the reason for their action and by the fact that there is no apparent way of contacting the person who runs the Planet to ask them about it. Needless to say this did not encourage me to write further posts about Fedora.

If a blog has to be removed from a feed due to technical reasons then the correct thing to do is to inform the blogger of why it’s removed and what needs to be fixed before it can be added again.

If a blog is not meeting the content criteria then I expect that in most cases the blogger could be convinced to write more content that matches the criteria and tag it appropriately. Having criteria for some aspects of blog quality and encouraging the bloggers to meet the criteria can only improve the overall quality.

Currently there is a Planet installation on debian.net being recommended which is based on Planet Debian, but with some blogs removed (with no information available publicly or on debian-private as to what the criteria are for removing the blogs in question). It seems to me that if it’s worth using Debian resources to duplicate the Planet Debian then it should be done in a way that benefits readers (EG by going to the Planet vs Universe model that Ubuntu follows), and that if blogs are going to be removed from the feed then there should be criteria for the removal so that anyone who wants their blog to be syndicated can make whatever changes might be necessary.

[1] http://planet.debian.org/
[2] http://planet.linux.org.au/
[3] https://wiki.ubuntu.com/PlanetUbuntu
[4] http://planet.perl.org/
[5] http://planet.gentoo.org/universe/

Software Development is a Team Sport

April 16, 2008 | etbe

Albert writes about software development and how much teamwork is used [1]. He makes an interesting clash of analogies by suggesting that it’s not a “team sport” because “its not like commercial fishing where many hands are used to pull in the net at the same time“.

I think that software development for any non-trivial project is a team sport. You don’t have the same level of direct coordination as required for pulling in a net (or the rugby scrum [2] to use a sporting analogy), but that doesn’t stop it being a team effort.

Some parts of team development projects are like a relay event, in corporate environments the work is done in parallel simply because everyone is working the same hours but in free software development projects the work is often serialised. I think that it’s often more effective to serialise some work, if someone is actively working on one sections of code it may save time to avoid working in that area until they are finished. There is little benefit in writing code to old interfaces.

Some parts of team projects have specialised skill areas (EG debugging, skills in particular programming languages, and graphical design). Soccer is one sport where different rules apply to different players (the goal keeper can use their hands). In ice-hockey the protective clothing used by the goal keeper is considerably different from that used by other players. In most team sports where the aim is to put a ball through a goal at one end (EG basketball and all versions of football) there seems to be some degree of specialisation, some players are dedicated to scoring goals while others are dedicated to defense. The fielding team in cricket has every player assigned to a different part of the field – with slight differences in the skills required.

Then there is the issue of large projects such as Linux distributions. It seems to me that a Linux distribution will always comprise multiple teams as well as some individual projects. Maybe we could consider Linux distributions (and distributions of the other free OSs) to be similar to countries that compete in the Olympics. The culture of the Free Software (or Open Source if that’s your idea) community can be compared to the Olympic Spirit. Of course the Olympic idea that people should come together in peace for the Olympic Games and that it’s about honor not money is pretty much dead.

Maybe the Free Software development processes should be compared to an ideal of what sporting contests would be if there weren’t unreasonable amounts of money (and therefore corruption) involved.

Of course no analogy is perfect and there are many ways in which this one breaks down. One of which is the cooperation between distributions. There is a lot of private discussion between developers of various distributions and upstream developers about how to plan new features. It’s not uncommon for developers to announce certain development decisions as soon as they are made to help other distributions make decisions – for a developer in a distribution project if there is an issue which doesn’t matter much to you or your users then it’s often good to strive for compatibility with other distributions.

When users advocate new features or changes they sometimes try multiple distributions. It’s not uncommon for a feature request to be rejected by one distribution and then accepted by another. Once a feature is included in a major distribution the upstream developer is more likely to accept it due to it’s wide testing. Then when the feature is in upstream it’s almost certain to be included in all other distributions. I often recommend that when someone disagrees with one of their bugs being closed as “not a bug” that they try reproducing it in another distribution and reporting it there. As a side note, the criteria for reporting a bug in any free software distribution is that you can describe it in a way that allows other people to reproduce it – whether it’s a bug that afflicts you every day or whether you installed the distribution for the sole purpose of reporting the bug in a new forum is not relevant. As a general rule I recommend that you not have the same bug report open in more than one distribution at any time (if you notice a bug reported in multiple distributions then please add a note to each bug report so that work can be coordinated). As a general rule the only situation where I will open the same bug in multiple forums is if I have been told that the responsible person or people in one forum are unwilling or unable to fix it.

Finally, the people who consider that they don’t need to be a team player because they do their coding alone might want to consider Qmail. Dan Bernstein is a great coder and Qmail is by most metrics a fine piece of software, in terms of security Qmail is as good as it gets! If Dan was more of a team player then I believe that his mail server would have been much more successful (in terms of the number of sites using it). However I do understand his desire to have a great deal of control over his software.

Blog Posts Should Stand Alone

April 6, 2008 | etbe

I believe that apart from some exceptions (such as “links” posts) each post should stand alone. A reader should be able to read a single blog post and understand the author’s point without needing to visit any external sites.

A common mistake is to write a post that can not be understood without following the links. This means that if one of the links gets taken down then the post can not be interpreted. Also if a reader has Internet access problems that deny access to the other site (which is not uncommon) they will be unable to find the original source and thus miss the point.

It’s quite common for people to download copies of blog content before going out of net access (I routinely load a Planet feed of the blogs I read before travelling). Some people read blog content via email, for such people reading blogs without net access will be even more common. If a blog post can’t be immediately understood then a significant number of readers will just skip it. If too many posts from one RSS feed (where “too many” is a subjective value that varies from reader to reader) have this problem then they may just unsubscribe from the feed.

Also even people who do have good net access will sometimes skip posts which require them to visit an external site. It takes more time and if they aren’t sure that the content will be of interest then they skip it.

Finally writing an explanation of your point tends to result in more clear communication. At the shallow end of the blog pool it’s quite common to see posts which link to web pages and express disagreement with them. If the web page which is referenced makes several points (it’s very rare to find pages which strictly make a single point with no sub-points and no chain of logic to support the point) then it can be difficult or impossible to determine what the blogger specifically disagreed with. A post which summarises a page and gives specific reasons for agreeing or disagreeing with it gives little potential for confusion or miscommunication.

SE Linux Play Machine and Passwords

April 2, 2008 | etbe

My SE Linux Play Machine [1] has been online again since the 18th of March.

On Monday the 11th of Feb I took it offline after a user managed to change the password for my own account (their comment was “ohls -lsa! i can change passwordls -lsals -lsa HACKED!“). Part of the problem was the way /bin/passwd determines whether it should change a password.

The previous algorithm (and the one that is currently used in Debian/Etch) is that if the UID of the account that is having it’s password changed doesn’t match the UID of the process that ran /bin/passwd then an additional SE Linux check is performed (to see if it has permission to change other user’s passwords). The problem here is that my Play machine has root (UID==0) as the guest account, and that according to the /bin/passwd program there is no difference between the root account (for unprivileged users) and the bofh account (which I use and which also has UID==0). This means of course that users of the root account could change the password of my account. My solution to this was to run chcon on the /bin/passwd program to give it a context that denied it the ability to change a password. The problem was that I accidentally ran the SE Linux program restorecon (which restores file contexts to their default values) which allowed /bin/passwd to change passwords, and therefore allowed a user to change the password of my account.

The semanage tool that allows changing the default value of a file context does not permit changing the default for a file specification that matches one from the system policy (so the sys-admin can’t override compiled in values).

I have now fixed the problem (the fix is in my Etch SE Linux repository [2] and has been accepted for Debian/Unstable and something based on it will go into the upstream branch of Shadow. See the Debian bug report #472575 [3] for more information.

The summary of the new code is that in any case where a password is not required to change the user’s password then SE Linux access checks will be performed. The long version is below:

The new algorithm (mostly taken from the Red Hat code base which was written by Dan Walsh) is that you can only change a password if you are running as non-root (which means that the pam_unix.so code will have verified the current password) or if you are running as root and the previous SE Linux security context of the process is permitted access to perform the passwd operation in the passwd class (which means it is permitted to change other user’s passwords).

The previous context (the context before one of the exec family of system calls was called) is used for such access checks because we want to determine if the user’s shell (or other program used to launch /bin/passwd) was permitted to change other user’s passwords – executing a privileged program such as /bin/passwd causes a domain transition and the context is different) than the program that was used to execute it. It’s much like a SETUID program calling getuid(2) to get the UID of the process which launched it.

To get the desired functionality for my Play Machine I don’t want a user to change their own password as the account is shared. So I appended password requisite pam_deny.so to the file /etc/pam.d/passwd (as well as the chfn and chsh commands) so that hostile users can’t break things. The new code in /bin/passwd will prevent users from taking over the machine if my PAM configuration ever gets broken, having multiple layers of protection is always a good thing.

The end result is that the Debian package and the upstream code base are improved, and my Debian Etch repository has the code in question.

etbe – Russell Coker

Linux, politics, and other interesting things

Tag Archives: Most Popular