2

Links May 2014

Charmian Gooch gave an interesting TED talk about her efforts to fight organised crime and corruption by prohibiting anonymous companies [1]. The idea of a company is to protect the owner from unlimited liability not to protect them from law enforcement.

Dr Nerdlove has an insightful article about sexual harassment in geek culture [2].

Rebecca Rose wrote an insightful article for Jezebel about the worse bullying advice ever, her summary is that it should be called “Ways We Can Get You Goddamn Kids to Act So We Never Have to Deal With Your Problems Ever” [3].

In the wake of the Heardbleed bug Imperial Violet wrote an informative article explaining why revocation checks isn’t the solution [4].

Martin Lukacs and Shiri Pasternak wrote an insightful article about the Canadian government’s attempts to stop Canadian aboriginies from exercising their legal rights [5]. I bet that the Australian government is doing the same things.

Nelson Groom wrote an interesting interview with Norrie May Welby, the first recognised agender person in Australia [6]. Marriage equality is bound to happen soon in Australia, now that the government officially recognises non-binary gender people it can’t refuse them the right to get married and therefore the straight/gay marriage distinction isn’t relevant.

BDA Technology has an interesting article on choosing fonts to make text more readable for dyslexic people [7].

Eamon Waterford wrote an informative article for the ABC about how early-intervention social policies save significant amounts of tax money [8]. Conservatives claim to want to save money and try to cut such programs which costs everyone in the long term.

Greta Christina wrote an insightful article for Salon about why religious people want atheists to lie and pretend to believe [9].

Chris Mooney wrote an informative URL about a machine to test whether someone is liberal or conservative [10]. Paul Rosenberg wrote an informative article on the link between conservatism, evil, and psychopathy [11]. The next logical step is to treat conservatism as a mental health problem.

Sociological Images has an interesting article by Jay Livingston about the hypocrisy of “conservative” tax policies [12].

Scientific American reprinted an article from The Conversation by Elaine McKewon about the climate deniers who intimidated a journal into retracting an article about their belief in conspiracy theories [13]. It seems obvious that the climate deniers are the ones who conspire.

Ben Caldecott wrote an interesting article for the ABC about the fossil fuel divestment campaign [14]. It seems that this is getting some success already, as renewable energy will soon be cheaper than coal power this could kill off coal.

Mark Taylor wrote an interesting blog post titled “Observations of an Internet Middleman” about the operations of Level3 and Internet peering [15]. He explains how monopoly Telcos throttle their customers’ Internet access.

Matt Savoy wrote an informative and disturbing article about the fact that US cops are twice as likely to beat their wives as the general population [16]. Apparently the police hierarchy aren’t interested in prosecuting such crimes.

Paul Rosenberg wrote an insightful article about mythos vs logos and the conservative approach to relity [17]. One interesting point he makes is that white men (and members of privileged groups generally) fear a loss of status more than more realistic concerns (such as global warming).

Nick wrote an interesting blog post about using GPG encrypted email on an Android phone [18]. I should get this going on my phone.

3

Why I Use BTRFS

I’ve just had to do yet another backup/format/restore operation on my workstation due to a BTRFS corruption problem, but as usual I didn’t lose any data. The BTRFS data integrity features work reasonably well even when the filesystem gets into a state where the kernel will only accept a read-only mount.

Given that the BTRFS tag on my blog is mostly about problems with BTRFS I think it’s time that I explain why I use it in spite of the problems before people start to worry about my sanity or competence.

The first thing to note is that BTRFS is fairly resiliant toward errors when mounted in read-only mode. When mounting a filesystem read-write there are a number of ways in which things can break which are often due to kernel code not being able to handle corrupt metadata – I don’t know how much of this is inherent to the design of BTRFS and how much is simply missing features in filesystem error handling. Some of the errors that I have had weren’t entirely the fault of BTRFS, I twice had to do a backup/format/restore of my workstation due to a faulty DIMM corrupting memory (that has the potential to mess up any filesystem), but I still didn’t lose any data AFAIK.

The next thing to note is that I don’t use BTRFS when doing paid sysadmin work. ZFS is a solid and reliable filesystem and it is working really well for my clients while BTRFS has too many issues at the moment. As an aside I’m not interested in any comments about the ZFS license situation from anyone who’s not officially representing Oracle.

I also don’t use BTRFS on systems that I can’t access easily. The servers I have running BTRFS are all within an hour’s drive from home, while driving for an hour on account of a kernel or filesystem error is really annoying it’s not as bad as dealing with a remote server where I have no direct access.

Reasons to Use BTRFS

The benefits of BTRFS right now are snapshots (which are good for a first-line backup) and the basic data integrity features. I’ve found these features to work well in real use.

According to the comparison of Filesystems Wikipedia page ZFS and BTRFS are the only general purpose filesystems (IE for disks not tapes, NVRAM, or a cluster) that support checksums for all data and compression. Given that ZFS license issues will never allow it to be included in the Linux kernel tree it seems clear that BTRFS is the next significant filesystem for Linux. More testing of BTRFS is a good thing, while there are a number of known problems that the developers are working on it seems that more testing is needed now to find corner cases. Also we need a lot of testing to find bugs related to interactions with other software.

I’ve recently filed bug reports against the Debian installer because it can’t install to a BTRFS RAID-1 (fortunately BTRFS supports changing to RAID-1 after installation) and because it doesn’t support formatting an existing BTRFS filesystem (the mkfs program needs a -f option in that case). I also sent in a patch for the magic database used by file(1) to provide more information on BTRFS filesystems (which is in Debian/testing but not Debian/Wheezy). These are the sorts of things you encounter when routinely using software that you don’t necessarily notice in basic testing.

As an aside the Debian installation process failed at the GRUB step when I manually balanced a filesystem to use RAID-1 while the Debian installation was in progress. I didn’t file a bug report because the best advice is to not mess with filesystems while the installer is running. I’ll do a lot more testing of this when the Debian installer supports a BTRFS RAID-1 installation.

A final thing that we need to work on is developing sysadmin best practices and scripts for managing BTRFS filesystems. I’ve done some work on scripts to create snapshots for online backups but there are issues of managing free space etc. But working out how to best manage a new filesystem is something that takes years because there are many corner cases you may only encounter after a system has been running for a long time. So I really wouldn’t want to be in the situation of using a new filesystem on an important server without having practice running it on less important systems, I did that with ZFS and now have a hacky first install that I have to support for years.

My First Ingress Anomaly

schwag from Ingress Anomaly

Ingress is an Android phone game developed by Google where you have to go to various places to capture “portals” which are based on landmarks or places of note. There’s a sci-fi back story which I haven’t cared much about (my local library has a heap of good sci-fi books I don’t have time to read). But if you are interested in low quality sci-fi you can start with the Youtube channel for Ingress [1]. But the best thing to do is to download the Ingress client program from the Google play store if you have an Android phone [2].

Periodically they have special events in the game called “Anomalies”, which usually involve many people competing to capture a small number of special portals. Usually a portal is fairly easy to capture so a high level player can easily capture a portal in at most a few minutes, but when there is a large group of players defending a portal by replacing the “resonators” as quickly as they are destroyed it becomes more challenging. I’m not going to try to explain this properly as I presume that readers either already know or don’t care. The Wikipedia page about Ingress has more information.

On Saturday there was an Anomaly where the main city was Melbourne. My team (Enlightened) won the event in Australia but apparently the event in the US at the same time was won by the other team (Resistance).

I had hoped that our event would be similar to some of the ones I’ve seen on the “Ingress Report” (a fictional news service on youtube) in which actors from the sci-fi videos about Ingress attended. But unfortunately it seems that the actors stay in the US.

Every player who was registered for the anomaly received a gift pack that contained three cards and a badge. The picture above shows the cards that my wife and I received along with one of our badges. The badges are green for Enlightened and blue for Resistance (the in-game colors) and have three flashing LEDs – the photo has one of the LEDs lit up. The three cards have “passcodes” which can be redeemed for in-game items, the XMP, Power Cubes, and Resonators weren’t that useful (10 not particularly rare items that I could get through 10 minutes of game playing). The “Medal” cards are for in-game badges showing the anomalies that we have played at, but they are “bronze” badges which again aren’t much use as nothing less than silver will help in leveling up.

Overall it was a good experience, we won, had a good after-party (with free snacks and one free drink per person), and the badges are nice. I recommend attending if there’s an event in your area, and note that you don’t need to be a high level player to attend, even L6 players were able to contribute.

5

Mobile Phones and Cognitive Costs

I previously wrote about the case for a Basic Income in Australia [1]. Since then I read an interesting article by Rutger Bregman in the Dutch site The Correspondent discussing the benefit that such Basic Income schemes have given in various places [2], he makes a great case for giving a Basic Income to poor people. However I believe that it should be provided to everyone. I believe that government payments shouldn’t be “means tested” because the proportion of the population who earn more than the cutoff is usually very small (IE it doesn’t save much money), because the money will be recovered from wealthy people in tax, and because the bureaucracy involved in determining who is eligible involves spending tax money to pay bureaucrats and costing everyone time in dealing with the process.

The concept of an Attention Economy has been getting some interest recently. But discussions of it seem to be mostly concerned with the needs of fairly wealthy people, how to advertise to them, how to filter out unwanted ads for them, etc. But the costs in attention can be even worse for people on low incomes or social security because they have many attention costs that could be solved by money. Many common essential purchases (EG soap, basic foods, and purely functional clothes) can become difficult decisions if saving a few dollars is important. The amount of thought involved in buying such items can result in someone who’s not well off having little energy to concentrate on other things. I think that to have an effective social security system we need to consider how much cognitive effort is being required of people who are receiving benefits.

Last month I wrote yet another review of mobile phone plans in Australia [3]. Every time I do this it takes a significant amount of time because the Telcos devise overly complex plans to make it difficult to compare their offerings. It seems that they aim to make their offerings somewhat competitive if you can interpret them but confusing enough to allow many (most?) of their customers to pay needlessly high bills. I could just pick a random telco and pay whatever it takes, spending an extra $10 or $20 per month isn’t such a big deal for me. Even though I advise a number of my relatives on the cheapest phone plan it’s probably only barely saving enough money to be worth the effort, a major motivation for me is being stubborn and not wanting telcos to rip people off.

The Importance of Mobile Phones

A mobile phone seems to be essential for most people. Mobile phones are used for making medical appointments and for applying for work, so in almost every case some of the financial assistance paid to a job seeker or person on a disability pension will go towards mobile phone costs. The aims of the social security benefits seem to clearly involve a phone service. Given that most people who receive social security payments spend some of that money on mobile phone bills the government has a direct financial interest in not having such money wasted on needlessly expensive phone plans. As we also don’t want to have people on social security spending time and attention trying to find the best phone plan it makes sense to have the government help manage this process.

Good communications is regarded as essential to education and economic development. A huge amount of taxpayer money was spent on the NBN (and then wasted by Tony Abbott when he cancelled the original plans) for these benefits.

Good phone access is also an issue of personal safety. While it is possible to call emergency services from a mobile phone that doesn’t have a valid SIM there are probably few people who keep a spare phone charged for that purpose. Also there are a variety of issues that can impact life or health which can best be resolved by calling friends or relatives.

Free Phones

I think that the best solution to these problems is for the government to directly pay mobile phone bills. If we are going to have a Basic Income then the government should make the base payment for mobile phone calls for every resident, if not then the government should make the base payment for everyone who receives any form of social security payment. Obviously the amount of data that could be transferred over a 3G network is greatly in excess of the amount that any plans allow and offering true “unlimited” calling leads to people using mobile phones as baby-monitors, so some limits are necessary. But the government could cheaply pay for a phone plan that exceeds the requirements of most people and then allow anyone who wants more data or calls to pay extra.

Which Plan?

The cheapest all-inclusive mobile phone plan in recent times was Kogan who charged $299 per annum for 6G of data per month and “unlimited” calls. The cheapest current offer for “unlimited” calls seems to be Lebara’s $30 per month offer which includes 2G of data.

My observation is that people who use 3G Internet just for phone use (as opposed to tethering a laptop or providing home Internet), who use a home Wifi network for installing Android programs and don’t play Ingress tend to use a lot less than 1G of data per month. I would use between 400M and 700M per month if I wasn’t playing Ingress and didn’t use my phone as a Wifi access point.

In terms of call volume, 600 minutes a month should be enough to cover the basic needs of most people judging by the number of plans on offer with limits smaller than that.

As a current Lebara offer is $30 per month for “unlimited” calls and 2G of data (and there are similar offers from other telcos), it should be possible for a telco to offer 600 minutes of calls and 1G of data for less than $15 per month. When the government is paying (no bad debts) and many of the users will use much less than the quota such a plan should be profitable at $10 per month or less.

Implementation

The way to implement this would be to start with a tender that has a hard requirement for the minimum amount of service provided. This would include the number of minutes per month for calls, the number of SMS sent, and the amount of data transfer. To avoid excessive billing (a standard telco scam) there should be a requirement that fees for extra use not exceed some small multiple of the base rate, for example if the government was paying $10 for 600 minutes of calls and 1G of data the excess usage charges could be capped at 3 cents per meg and 5 cents per minute (3* the base rate) which would be profitable for the telco but not a scam.

If the telcos act in a market driven competitive manner they will compete to provide the most attractive offerings for $10 per month.

If the build cost of the NBN had been invested at government bond rates (for any of the rates used for current bonds) then it would pay for the mobile phone bills for every resident forever, so the cost of providing phones for everyone is in the range of government telecommunications programs. If people who are working were to pay an extra $10 in tax and people who are on social security had their benefits reduced by $10 then that would be a great deal if it saved them the $20 or more that most people spend on a basic phone plan.

Government Phone Delays

A friend who’s a single mother pays for an “unlimited” mobile phone plan because calls to the social security office are usually kept on hold for more than an hour. It’s likely that she could save money over the course of a year by paying for a cheaper plan and just dealing with the occasional big bill, but she needs to budget carefully and feels that she can’t take that risk. It seems stupid that government phone delays (partly caused by cutting funds for staff) cause increased expenses for people who are receiving social security payments. If the government implemented a smart queuing system that allowed them to call people back instead of making them wait on hold then it would save money overall.

So there are obviously other ways that the government can save money on phone calls and time for everyone.

3

BTRFS vs LVM

For some years LVM (the Linux Logical Volume Manager) has been used in most Linux systems. LVM allows one or more storage devices (either disks, partitions, or RAID sets) to be assigned to a Volume Group (VG) some of which can then allocated to a Logical Volume (LVs) which are equivalent to any other block device, a VG can have many LVs.

One of the significant features of LVM is that you can create snapshots of a LV. One common use is to have multiple snapshots of a LV for online backups and another is to make a snapshot of a filesystem before making a backup to external storage, the snapshot is unchanging so there’s no problem of inconsistencies due to backing up a changing data set. When you create a snapshot it will have the same filesystem label and UUID so you should always mount a LVM device by it’s name (which will be /dev/$VGNAME/$LVNAME).

One of the problems with the ReiserFS filesystem was that there was no way to know whether a block of storage was a data block, a metadata block, or unused. A reiserfsck --rebuild-tree would find any blocks that appeared to be metadata and treat them as such, deleted files would reappear and file contents which matched metadata (such as a file containing an image of a ReiserFS filesystem) would be treated as metadata. One of the impacts of this was that a hostile user could create a file which would create a SUID root program if the sysadmin ran a --rebuild-tree operation.

BTRFS solves the problem of filesystem images by using a filesystem specific UUID in every metadata block. One impact of this is that if you want to duplicate a BTRFS filesystem image and use both copies on the same system you need to regenerate all the checksums of metadata blocks with the new UUID. The way BTRFS works is that filesystems are identified by UUID so having multiple block devices with the same UUID causes the kernel to get confused. Making an LVM snapshot really isn’t a good idea in this situation. It’s possible to change BTRFS kernel code to avoid some of the problems of duplicate block devices and it’s most likely that something will be done about it in future. But it still seems like a bad idea to use LVM with BTRFS.

The most common use of LVM is to divide the storage of a single disk or RAID array for the use of multiple filesystems. Each filesystem can be enlarged (through extending the LV and making the filesystem use the space) and snapshots can be taken. With BTRFS you can use subvolumes for the snapshots and the best use of BTRFS (IMHO) is to give it all the storage that’s available so there is no need to enlarge a filesystem in typical use. BTRFS supports quotas on subvolumes which aren’t really usable yet but in the future will remove the need to create multiple filesystems to control disk space use. An important but less common use of LVM is to migrate a live filesystem to a new disk or RAID array, but this can be done by BTRFS too by adding a new partition or disk to a filesystem and then removing the old one.

It doesn’t seem that LVM offers any benefits when you use BTRFS. When I first experimented with BTRFS I used LVM but I didn’t find any benefit in using LVM and it was only a matter of luck that I didn’t use a snapshot and break things.

Snapshots of BTRFS Filesystems

One reason for creating a snapshot of a filesystem (as opposed to a snapshot of a subvolume) is for making backups of virtual machines without support from inside the virtual machine (EG running an old RHEL5 virtual machine that doesn’t have the BTRFS utilities). Another is for running training on virtual servers where you want to create one copy of the filesystem for each student. To solve both these problems I am currently using files in a BTRFS subvolume. The BTRFS kernel code won’t touch those files unless I create a loop device so I can only create a loop device for one file at a time.

One tip for doing this, don’t use names such as /xenstore/vm1 for the files containing filesystem images, use names such as /xenstore/vm1-root. If you try to create a virtual machine named “vm1” then Xen will look for a file named “vm1” in the current directory before looking in /etc/xen and tries to use a filesystem image as a Xen configuration file. It would be nice if there was a path for Xen configuration files that either didn’t include the current directory or included it at the end of the list. Including the current directory in the path is a DOS mistake that should have gone away a long time ago.

Psychology and Block Devices

ZFS has a similar design to BTRFS in many ways and has some similar issues. But one benefit for ZFS is that it manages block devices in a “zpool”, first you create a zpool with the block devices and after that you can create ZFS filesystems or “ZVOL” block devices. I think that most sysadmins would regard a zpool as something similar to LVM (which may or may not be correct depending on how you look at it) and immediately rule out the possibility of running a zpool on LVM.

BTRFS looks like a regular Unix filesystem in many ways, you can have a single block device that you mount with the usual mount command. The fact that BTRFS can support multiple block devices in a RAID configuration isn’t so obvious and the fact that it implements equivalents to most LVM functionality probably isn’t known to most people when they start using it. The most obvious way to start using BTRFS is to use it just like an Ext3/4 filesystem on an LV, and to use LVM snapshots to backup data, this is made even more likely by the fact that there is a program to convert a ext2/3/4 filesystem to BTRFS. This seems likely to cause data loss.

2

Source Escrow for Proprietary Software

British taxpayers are paying for extra support for Windows XP due to a lack of planning by the UK government [1]. While the cost of this is trivial compared to other government stupidity (such as starting wars of aggression) this sort of thing should be stopped.

The best way to solve such problems is for governments to only use free software. If the UK government used Red Hat Enterprise Linux then when Red Hat dropped support for old versions they would have the option of providing their own support for old versions, hiring any other company to support old versions, or paying Red Hat for supporting it. In that case the Red Hat offer would probably be quite reasonable as competition drives the prices down.

It doesn’t seem likely that the UK government will start using only free software in the near future. It’s not impossible to do so, there are organisations dedicated to this task such as Free-gov.org which aims to develop e-government software that is under GPL licenses [2]. The Wikipedia page List of Linux Adopters [3] has a large section on government use, while not all entries are positive (some have reverted) it shows that it’s possible to use Linux for all areas of government. But governments often move slowly and in the case of wealthy countries such as the UK it can be easier to just tax the citizens a little more than to go to the effort of saving money.

But when governments use proprietary software they shouldn’t be restricted in support. It seems that the only way to ensure that the government can do what it needs is to have a source escrow system. Then if the company that owned the software ceased supporting it anyone who wanted to offer support would be able to do so. This would probably require that software which is out of support be released to the public domain so that anyone who wanted to tender for such support work could first inspect the code to determine if they were capable of doing the work.

People who believe the myths about secret source software claim that allowing the source code to be released would damage the company that owns it. This has been proved incorrect by the occasions when source code for software such as MS-Windows has been released on the Internet with no apparent harm. Also Microsoft have a long history of licensing their source code to universities, governments, and other companies for various purposes (including porting Windows to other CPUs). It’s most likely that some part of the UK government already has the full source code to Windows XP, and it’s also quite likely that computer criminals have obtained copies of the source by now for the purpose of exploiting security flaws. Also they stop supporting software when they can’t make money from providing the usual support, so by definition the value to a company of the copyright is approaching zero by the time they decide to cease support.

Given the lack of success experienced by companies that specialise in security (for example the attack on RSA to steal the SecurID data [4]) it doesn’t seem plausible that Microsoft has had much success in keeping the source to Windows XP (or any other widely used product) secret over the course of 12 years.

In summary source code to major proprietary software products is probably available to criminals long before support expires and is of little value to the copyright owners. But access to it can provide value to governments and other users of the software.

The only possible down-side to the software vendor is if the new version doesn’t provide any benefits to the user. This could be a problem for Microsoft who seem to have the users hate every second version of Windows enough to pay extra for the old version. The solution is to just develop quality software that satisfies the needs of the users. Providing a legal incentive for this would be a good idea.