Blogging and Documents

September 30, 2007 | etbe

It seems that the majority of blog traffic (at least in blogs I read) is time-based. It is personal-diary entry posts, references to current events, predictions about future events, bug reports, and other things that either become obsolete or for which it’s important to know the date. For such posts it makes sense to have the date be part of the Permalink URL, and in the normal course of events such posts will tend not to be updated after release.

Another type of blog traffic is posts that have ongoing reference value which will (ideally) be actively maintained to keep them current. For such posts it makes sense to have no date stamp in the Permalink – for example if I update a post about installing SE Linux on Etch once Lenny is released (a significant update) I don’t want people ignoring it when it comes up in search engines (or worse having search engines score it down) because the URL indicates that it was written some time before the release of Etch.

WordPress supports Pages as separate entities to Posts, and the names of Pages are direct links under the root of the WordPress installation. However there is no RSS feed for Pages (AFAIK – I may have missed something) and the WordPress themes often treat Pages differently (which may not be what you want for timeless posts). Also it is not unreasonable to have Pages and timeless posts.

I’m thinking of creating a separate WordPress installation for posts that I intend to manage for long periods of time with updates (such as documenting some aspects of software I have written). The management options for a blog server program provide significant benefits over editing HTML files. The other option would be to use a different CMS (a blog server being a sub-category of a CMS) to store such things.

What I want is a clear way of presenting the data with minimal effort from me (an advantage of WordPress for this is that I have already invested a significant amount of effort in learning how it works) and editing from remote sites (the offline blog editing tools that are just coming out is a positive point for using a blog server – particularly as I could use the same editor for blog posts and documents).

Any suggestions as to how to do this?

Then of course there’s the issue of how to syndicate this. For my document blog (for want of a better term) I am thinking of updating the time-stamp on a post every time I make a significant change. If you subscribe to the document feed than that would be because you want to receive new copies of the documents as they are edited. The other option would be to not change the time-stamp and then include the feed along with my regular blog feed (making two feeds be served as one is not a technical challenge). If I was to update the time stamps then I would have to write posts announcing the release of new documents.

Does anyone know of someone who writes essays or howto documents in a similar manner to Rick Moen [1] or Paul Graham [2] who also does daily blog posts? I’d like to see some examples of how others have solved these problems (if there are any).

[1] http://linuxmafia.com/
[2] http://www.paulgraham.com/

Upgraded to WordPress 2.3

September 29, 2007 | etbe

I just upgraded to WordPress 2.3. When using Konqueror (my favourite browser) the comment approval is slightly broken (when I tag a comment as spam it usually just turns red and doesn’t disappear from the main Comments tab) and I have to refresh that window more often than usual to make sure I got the result I desired. Also the Sidebar Widget editing is totally broken in Konqueror, I guess I’ll have to login with Firefox to get the new tags feature working.

Also I have got a few WordPress errors about the table “$table_prefix” . “post2cat” not existing. The table in question doesn’t exist in any of my blogs.

So far this is the worst WordPress upgrade experience I’ve had (I started on 2.0).

Swap Space

September 28, 2007 | etbe

There is a wide-spread myth that swap space should be twice the size of RAM. This might have provided some benefit when 16M of RAM was a lot and disks had average access times of 20ms. Now disks can have average access times less than 10ms but RAM has increased to 1G for small machines and 8G or more for large machines. Multiplying the seek performance of disks by a factor of two to five while increasing the amount of data stored by a factor of close to 1000 is obviously not going to work well for performance.

A Linux machine with 16M of RAM and 32M of swap MIGHT work acceptably for some applications (although when I was running Linux machines with 16M of RAM I found that if swap use exceeded about 16M then the machine became so slow that a reboot was often needed). But a Linux machine with 8G of RAM and 16G of swap is almost certain to be unusable long before the swap space is exhausted. Therefore giving the machine less swap space and having processes be killed (or malloc() calls fail – depending on the configuration and some other factors) is probably going to be a better situation.

There are factors that can alleviate the problems such as RAID controllers that implement write-back caching in hardware, but this only has a small impact on the performance requirements of paging. The 512M of cache RAM that you might find on a RAID controller won’t make that much impact on the IO requirements of 8G or 16G of swap.

I often make the swap space on a Linux machine equal the size of RAM (when RAM is less than 1G) and be half the size of RAM for RAM sizes from 2G to 4G. For machines with more than 4G of RAM I will probably stick to a maximum of 2G of swap. I am not convinced that any mass storage system that I have used can handle the load from more than 2G of swap space in active use.

The reason for the myths about swap space size are due to some old versions of Unix that used to allocate a page of disk space for every page of virtual memory. Therefore having swap space less than or equal to the size of RAM was impossible and having swap space less than twice the size of RAM was probably a waste of effort (see this reference [1]). However Linux has never worked this way, in Linux the virtual memory size is the size of RAM plus the size of the swap space. So while the “double the size of RAM” rule of thumb gave virtual memory twice the size of physical RAM on some older versions of Unix it gave three times the size of RAM on Linux! Also swap spaces smaller than RAM have always worked well on Linux (I once ran a Linux machine with 8M of RAM and used a floppy disk as a swap device).

As far as I recall some time ago (I can’t remember how long) the Linux kernel would by default permit overcommitting of memory. For example if a program tried to malloc() 1G of memory on a machine that had 64M of RAM and 128M of swap then the system call would succeed. However if the program actually tried to use that memory then it would end up getting killed.

The current policy is that /proc/sys/vm/overcommit_memory determines what happens when memory is overcommitted, the default value 0 means that the kernel will estimate how much RAM and swap is available and reject memory allocation requests that exceed that value. A value of 1 means that all memory allocation requests will succeed (you could have dozens of processes each malloc 2G of RAM on a machine with 128M of RAM and 128M of swap). A value of 2 means that a different policy will be followed, incidentally my test results don’t match the documentation for value 2.

Now if you run a machine with /proc/sys/vm/overcommit_memory set to 0 then you have an incentive to use a moderately large amount of swap, safe in the knowledge that many applications will allocate memory that they don’t use, so the fact that the machine would deliver unacceptably low performance if all the swap was used might not be a problem. In this case the ideal size for swap might be the amount that is usable (based on the storage speed) plus a percentage of the RAM size to cater for programs that allocate memory and never use it. By “moderately large” I mean something significantly less than twice the size of RAM for all machines less than 7 years old.

If you run a machine with /proc/sys/vm/overcommit_memory set to 1 then the requirements for swap space should decrease, but the potential for the kernel to run out of memory and kill some processes is increased (not that it’s impossible to have this happen when /proc/sys/vm/overcommit_memory is set to 0).

The debian-administration.org site has an article about a package to create a swap file at boot [2] with the aim of making it always be twice the size of RAM. I believe that this is a bad idea, the amount of swap which can be used with decent performance is a small fraction of the storage size on modern systems and often less than the size of RAM. Increasing the amount of RAM will not increase the swap performance, so increasing the swap space is not going to do any good.

Eating Corpses

September 27, 2007 | etbe

Davyd Madeley writes about vegetarianism for the environment [1] which is listed in Wikipedia as Environmental Vegetarianism [2]. He links to an article on the Huffington Post [3]. The Huffington Post article in turn links to an article on GoVeg.com about global warming [4].

Mass-produced meat is not only bad for the environment but there are also health issues related to meat consumption (due to bad practices in mass farming, combining the meat of thousands of animals into mince thus increasing the spread of bad meat, and the fact that most people in first-world countries consume significantly more meat than anyone did at any time in history).

One thing that doesn’t get mentioned in these posts is the fact that farming is not required to produce meat. In fact the meat that is most healthy (due to lack of carcinogenic chemicals and free-range feeding) and has the strongest flavour (which may be a good or bad thing depending on whether you actually like the flavour of meat) is from wild animals. If you don’t like the taste of meat (which seems to be the case when people don’t like game meat) then why eat it at all?

In Australia large numbers of kangaroos are killed because they eat grass more efficiently than cattle (they have evolved over tens of thousands of years to survive in Australian conditions unlike cattle). There are also a number of foreign animals that have run wild in Australia and are considered vermin, this includes rabbit, pig, buffalo, deer and camel (all of which are tasty).

Even among native animals there are often times when a cull is needed. If some good seasons allow the population to increase then when there is a bad season the population has to reduce and it’s often better for them to be culled (thus providing plenty of food for the surviving animals) than for all of them to starve.

There is a game meat wholesaler I’ve visited a few times that sells buffalo, rabbit, pig, camel, crocodile, possum, emu, kangaroo, and some other animals. All of the meat is from wild animals (apart from rabbit and pig none of those animals can be domesticated). I’m sure that every region has such a wholesaler that will sell to interested individuals if you know where to look (it seems impossible to buy any game meat other than kangaroo retail in Australia).

Finally one thing that offends me is people who eat meat but are not prepared to kill the animal. If you aren’t prepared to kill it then you shouldn’t pay someone else to do so on your behalf! Claiming that “the animal was going to be killed anyway” is a pitiful excuse that is only suitable for children. It’s acceptable for children to eat meat without thinking about where it came from. But adults should be able to deal with the fact that eating meat means killing animals – or become vegetarian if they can’t cope with it.

The book 3001 The Final Odyssey pioneered the term “corpse food” for eating meat. I believe that the term is accurate and should be used. If you can’t stomach eating corpses then there are many good vegetarian options available.

There are many vegetarians in the Linux community. As these issues are getting discussed a lot recently maybe it would be good to have the vegetarians choose some good vegetarian restaurants to have Linux meetings on occasion. Davyd got a bit of negative feedback on his post, maybe if he invited a bunch of his local Linux people to have dinner at a vegetarian restaurant and they enjoyed the food then the reaction to such ideas would be more positive.

Is the Server Dead?

September 26, 2007 | etbe

I often get reports such as “the server was dead so I rebooted it“. This really doesn’t help me fix the problem, so if the person who uses the server wants reliability (and doesn’t want to be rebooting it and losing data all the time) then more information needs to be provided. Here is a quick list of tests to perform before a reboot if you would like your server not to crash in future:

Does pressing the CAPS-LOCK key on the keyboard make the CAPS LED light up? If so then the OS isn’t entirely dead.
What is on the screen of the server (you may have to press a key to get the screen to un-blank)? If it’s a strange set of numbers then please photograph them if possible, I might understand what they mean. If you don’t have a camera with high enough resolution to capture them then please make a note of some of the messages. Don’t write down numbers – they are not useful enough to be worth the effort. Write down words, including special words such as OOM and pairs of words seperated by a “_” character.
If the “server” is a Xen virtual machine then save the contents of the console (as described in my previous post [1]).
Can you ping the machine (usually by ping servername)? If so then networking is basically operational.
Are the hard drive access lights indicating heavy use? If so then it might be thrashing due to excessive memory use (maybe a DOS attack).
Can you login at the console? If so please capture the output of free, ps auxf, and netstat -tn.
If the machine offers TCP services (almost all servers do) then use the telnet command to connect to the service port and make a note of what happens. For example to test a mail server type “telnet server 25” and if all goes well you expect to see “220 some message from the mail server“, note how long it takes for such a message to be displayed. Some protocols don’t send a message on a connect, for example with HTTP (the protocol used by web servers) you have to enter a few characters and press ENTER to get a response (usually some sort of HTTP error message).

Finally please don’t tell me that the server is too important and that the users couldn’t wait for you to perform any tests before rebooting it. If the server is important then it is important that it doesn’t crash repeatedly. A crash may even be caused by something that could cause data loss (EG hardware that is failing) or something that could incur extra expense if not fixed quickly (EG failing hardware that will be out of warranty soon). You have to tell users that the choice is to wait for an extra few minutes or risk having another crash tomorrow with further data loss.

If the server is important enough for it to be worth my time to try and fix it then it’s important enough to have these tests performed before the reboot.

[1] http://etbe.coker.com.au/2007/07/27/a-support-guide-to-xen/

Census

September 25, 2007 | etbe

I’m just completing Jeff and Pia Waugh’s Australian Open Source Industry & Community Census [1]. There are some things that can be improved with that survey in particular and surveys in general.

It seems to be assumed that everyone is trying to work full-time. I admit that there are probably few people who have decided that they don’t need as much money as they would earn in full-time work and reduced their work hours to match their financial needs (as I have done). Surveys that just ask a for a figure of how much is earned add to the pressure to earn as much as possible, which isn’t what’s best for most people.

I have the impression that the questions about “how long have you been doing X” assume that doing so is contiguous. If that’s the case then asking “when did you first do X” and giving a drop-down list-box of the last 20 years to select from would probably be better (more precise and remove confusion).

Debian wasn’t listed as a Linux distribution! It only has Ubuntu, RHEL, Fedora, and SUSE.

The question mixing kernels and distributions was a little strange. It gives BSD as a kernel option but no BSD option in user-space, so apparently it’s assumed that there is only one user-space/distribution for BSD (there have been some attempts to release Debian with the various BSD kernels). Also it wasn’t clear what you have to do regarding the kernel to select it (is the couple of hundred lines of patches I submitted to the Linux kernel adequate to let me list it?).

There were a bunch of questions that would get very predictable answers. Do you want to have access to official government web-sites and documents with free software? I guess that they want to get a large number of people requesting such things. Incidentally much of that is in the Greens IT policy…

Immediately below the buttons on the screen to go to the Next and Previous pages there is a link to clear the form and exit. When doing something difficult like sys-admin work it’s expected that a command to wipe out your work will be immediately next to something innocuous, but for something that generally doesn’t require much concentration (such as filling out a survey) it would be good to have the dangerous options a little further away.

At the end of the survey there are questions about whether you want to be contacted about events held by various companies. I think that it would be better to have an RSS feed of such events that we can poll when we have spare time. I’m sure that the PR people running the events are happy when they see a good number of people signed up to their mailing list. But if you actually want to get people to sign up without prior contact the best thing to do is have it on a web site with an RSS feed (either a blog or a CMS) so that it can be polled, syndicated, and googled.

[1] http://waughpartners.com.au/research/census2007″

Blogger is Not for Serious Blogging

September 24, 2007 | etbe

When I started blogging I used Blogger [1]. After some time I decided that it did not offer me the freedom I desired. I could not easily make changes (I could have created new themes, but it would have taken an unreasonable amount of work). I currently use WordPress, it’s still a lot of work to change themes, but at least it’s CSS and PHP coding which can be used for other things. Blogger offers no statistics on web use (I tried adding support for Google Analytics but couldn’t get it to work properly), what I want is Webalizer or something similar (which is easy to do when running your own server).

Blogger is a reasonable way of starting blogging, but if you use it then you want to make it easy to change to something that you own. Blogger has a feature of using a DNS name in a domain that you own for the name of your blog (which is much less obvious than it once was), I regret not using that feature as I still have my old posts on blogger and don’t want to break the links.

Blogger has in the past had problems with time-stamps on posts, when I used blogger I had some complaints that my posts were staying at the top of Planet listings for unreasonable amounts of time (I never tracked this down before switching to my own platform).

Hosting your own blog is not as difficult as you might expect (initially at least). It becomes difficult when you want to install lots of plug-ins, but then any blogging solution would be difficult if you want to do that. The WordPress [2] package in Debian works well and has good support for multiple WordPress blogs. There is a separate product named WordPress-MU [3] which is designed for people who want to run a service in competition with Blogger, some people recommend that you use WordPress-MU if you want to set up blogs for several people. I disagree. If you are setting up blogs for a small number of people then you can use the standard WordPress package and create a file named /etc/wordpress/config-whatever.example.com.php which contains the configuration for whatever.example.com and then create a new WordPress blog by using the web-based interface to do the rest. It would not be difficult to create the configuration file in question with an M4 script if you have a moderate number of blogs to host (maybe a hundred or so). I think that it’s only if you want to host thousands of blogs that you need the features of WordPress-MU. Note that MU is not as well packaged as the base WordPress and has some rough edges. Last time I tried to set up MU I was not successful.

This is not to say that WordPress is inherently the best program, there are many other free software blogging platforms out there. WordPress is the one that I use and am happy to recommend but if your requirements differ from mine then another platform may be better for you. I also suggest that WordPress be used as the base-line for comparing blogging software.

Blogger does not require significant resources. A virtual host with 256M of RAM should be more than adequate to run WordPress plus MySQL. Such virtual hosts are getting quite cheap nowadays, and one such host could easily be shared by a number of bloggers. My blog uses about 1.2G of data transfer per month. vpsland.com offers virtual hosts starting at 150G per month data transfer with 192M of RAM being the minimum. Prices start at $US15 per month. While I can’t compare vpsland.com to other virtual hosting providers (having never used any other such service) I can say that they work reasonably well and I have a client who is happy with them. So it seems that a minimal plan with vpsland.com would host 20 blogs with the same traffic as mine (with RAM being the limiting factor) and a slightly larger plan (with more RAM and more bandwidth) that costs $US30 or $US40 per month could handle 100 or more blogs that are similar to mine. If you get together with some friends and share a virtual server then blogging would not be expensive. Incidentally I had previously read a blog comment about people being hesitant to share servers with their friends (as they apparently would rather grant some unknown people at a faceless corporation the ability to snoop on them than people that they know). The advantage of a blog server in this regard is that everything is public anyway!

If you have good technical skills then I recommend using WordPress as your first blogging platform. If you find that you don’t like it for some reason then you can convert to another platform if you own the domain. If you are setting up a blog for a less technical user then WordPress is also a good choice. My sister uses WordPress, not that she made much of a choice (I had set up a Blogger account for her some time ago which she never used – I guess that could be considered as a choice to not use Blogger) but that I set up a WordPress blog for her and she seemed to like using it.

DMCA and Sci-Fi

September 23, 2007 | etbe

My previous post about the SFWA falsely issuing DMCA take-down notices [1] has got some reactions, many of which indicate a lack of clear reading of my post.

I am not advocating boycotting sci-fi. I am merely changing the priorities for my reading list. There must be at least 10 sci-fi books on my shelf of books to read, so refraining from buying more for a while isn’t going to impact my reading much either.

In regard to whether boycots are good or bad, it should be considered that the enjoyment of a work of art is highly subjective. If you don’t like the artist then you probably won’t enjoy the art. Boycots on products for which there are objective criteria of quality are often unsuccessful because the buyers have to make a conscious decision to buy a product of a lower quality or a higher price. I can’t make any objective claim about the relative merits of the work of Cory Doctorow and Jerry Pournelle. But it is impossible for me to enjoy reading Jerry’s work as much as I enjoy reading Cory’s work due to my opinion of the two authors.

I have just read a post by Eva Whitley [2] (the widow of Jack L. Chalker – one of my favourite authors when I was younger). She starts by describing Jack’s attitude towards electronic publishing, which sounds quite enlightened. She then notes that Del Rey has the rights to some of his books which are officially still in-print but in practice impossible to get, Baen is still paying royalties but had paid him a large advance so he won’t receive money until the royalties exceed the advance, and some books are out of print but no publisher wants to buy the rights to do a re-print. So it seems that she is not receiving money from her late husbands work partly due to book companies being uncooperative (not printing the books and not permitting others to print them without payment) and partly due to being paid in advance.

Eva states that she is happy that the SFWA issued take-down requests for copies of the work of her late husband. That is good and in fact it’s a legal requirement. No-one can legally get copyright laws enforced unless they own the copyright or they are acting on behalf of the copyright owner. She is entitled to authorise the SFWA (or any other group) to act on her behalf in regard to her copyright violations. Neither the SFWA nor anyone else can take action against copyright violations without such permission.

Jerry Pournelle has written about the situation [3]. He complains that the site owners wanted the items listed individually. The Wikipedia page about the DMCA [4] makes it quite clear that you must provide such detailed information to get something taken offline, and that the DMCA take-down request MUST include a statement claiming ownership of the material UNDER PENALTY OF PERJURY.

If you make a false DMCA claim then you are committing perjury, according to Wikipedia the penalty for perjury in the US is up to five years in prison [5]. Sure any idiot can write up DMCA take-down requests for random stuff on the net and get them acted on (here’s an example of a 15yo idiot who did just that [6]). But if you want to stay out of jail you have to avoid making such false claims.

Jerry expresses a total lack of sympathy for Cory Doctorow and other people who have been victims of slander of title [7]. I wonder how he would react if someone started making public statements under penalty of perjury claiming that he didn’t own the title to some of his work. I suspect that he would desire something similar to what he desires to be done to people associated with 9-11 [8].

Finally one thing that I suggest doing to make some additional money from writing sci-fi is to release T-shirts. A basic T from cafepress.com was $7 last time I checked. $11 for a shirt based on a sci-fi book is not overly expensive and gives $4 profit for the author (more than twice what most authors make from a book sale). Shirt sales are unlikely to make as much money as book sales due to lower volumes, but the effort involved in creating a shirt design is not so great. A publishing company may deny an author (or their estate) future revenue by refraining from printing further copies, but unless they also own the trademarks related to the book and deny the author (or their estate) the right to use them then it should still be possible to make money from merchandise. Making money from merchandise is not as glamourous as making money from book royalties, but it can be effective as demonstrated by xkcd.com.

Context of /dev/xvc0

September 23, 2007 | etbe

I have just converted a Fedora Core 5 server to a CentOS 5 Xen Dom0 with Fedora Core 5 as a DomU.

The process took a little longer than expected because I didn’t have console or network access to the DomU initially. It turned out that /etc/modprobe.conf was configured to have the tg3 device for Ethernet while I really needed xennet to get networking going.

The console problem was due to the fact that the device /dev/xvc0 is used for the console in DomU’s and the Fedora Core 5 image was configured for a non-Xen mode of operation. Incidentally it seems a little strange that a default install of CentOS as a DomU will get gettys for /dev/tty[1..6] when none of them seem accessible. After I changed the /etc/inittab file to get the correct name it still didn’t work. It seems that the SE Linux policy in Fedora Core 5 doesn’t have the correct context for the /dev/xvc0 device.

semanage fcontext -a -f -c -s system_u -t tty_device_t /dev/xvc0

So I had to run the above semanage command to change the policy configuration, followed by restorecon /dev/xcv0 to apply the change (although once the change is configured it will apply after the next reboot).

Multiple DNS Names

September 21, 2007 | etbe

There are many situations where multiple DNS names for a single IP address that runs a single service are useful. One common example is with business web servers that have both www.example.com and example.com being active, so whichever a customer hits they will get the right content (the last thing you want is for a potential customer to make some trivial mistake and then give up).

Having both DNS names be equal and separate is common. One example of this is the way http://planet.ubuntulinux.org/ and http://planet.ubuntu.com/ both have the same content, it seems to me that planet.ubuntu.com is the more official name as the wiki for adding yourself to the Planet is wiki.ubuntu.com. Another example of this is the way http://planet.debian.org/ and http://planet.debian.net/ both have the same content. So far this month I have had 337 referrals to my blog from planet.debian.org and 147 from planet.debian.net. So even though I can’t find any official reason for preferring one over another the fact that more than 2/3 of the referrals from that planet come from the planet.debian.org address indicates that most people regard it as the canonical one.

In times past there was no problem with such things, it was quite routine to have web servers with multiple names and no-one cared about this (until of course one name went away and a portion of the user-base had broken links). Now there are three main problems with having two names visible:

Confusion for users. When a post on thedebianuser.org referred to my post about Planet Ubuntu it used a different URL to the one I had used. I was briefly worried that I had missed half (or more) of the content by getting my links from the wrong blog – but it turned out that the same content was on both addresses.
More confusing web stats for the people who run sites that are referenced (primarily the bloggers in the case of a Planet installation). This also means a lower ranking as the counts are split. In my Webalizer logs planet.debian.org is in position #5 and planet.debian.net is in position #14. If they were combined they would get position #3. One thing to keep in mind is that the number of hits that you get has some impact on the content. If someone sees repeated large amounts of traffic coming from planet.debian.org then they are likely to write more content that appeals to those users.
Problems with sites that have strange security policies. Some bloggers configure their servers to only serve images if the referrer field in the HTTP protocol has an acceptable value (to prevent bandwidth theft by unethical people who link to their pictures). My approach to this problem is reactive (I rename the picture to break the links when it happens) because I have not had it happen often enough to do anything else. But I can understand why some people want to do more. If we assume that an increasing number of bloggers do this, it would be good to not make things difficult for them by having the smallest possible number of referrer URLs. It would suck for the readers to find that planet.debian.org has the pictures but planet.debian.net doesn’t.

The solution to this is simple, one name should redirect to the other. Having something like the following in the Apache virtual host configuration (or the .htaccess) file for the least preferred name should redirect all access to the other name.

RewriteCond %{REQUEST_URI} ^(.*$) [NC]
RewriteRule . http://planet.example.com/%1 [R=301,L]

In my posts last night I omitted the URLs for the Planet Searches from the email version (by not making them human readable). Here they are:

Search Planet Linux Australia: http://google.com/coop/cse?cx=008627835083532581857%3A7j_t83ytzca
Search Planet Debian: http://google.com/coop/cse?cx=008627835083532581857%3Ak8pyp2qk8lm
Search Planet Ubuntu: http://google.com/coop/cse?cx=008627835083532581857%3Acexvbhdg9k8

etbe – Russell Coker

Linux, politics, and other interesting things

Monthly Archives: September 2007