etbe - Russell Coker

ZFS on Debian/Wheezy

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable.

BTRFS vs ZFS

Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don’t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don’t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity.

I am about to install a system to be a file server and Xen server which probably isn’t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won’t be upgraded a lot I’ll start with Debian/Wheezy even though it isn’t released yet because the system will be in use without much change well after Squeeze security updates end.

ZFS on Wheezy

Getting ZFS to basically work isn’t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn’t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn’t particularly difficult but it’s a pity that the zfsonlinux site didn’t support recent kernels.

Root on ZFS

The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device.

Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn’t ideal, but when you have an OS image that is backed up (and not so important) and a data store that’s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it.

There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort.

The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn’t work with the Debian initramfs for some reason and modules failed to load. This wasn’t necessarily a show-stopper as I can modify such things myself, but it’s another painful thing to manage and another way that the system can potentially break on upgrade.

The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the “zfs set mountpoint” command. This of course means that it doesn’t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn’t a good option. So for this new server I’ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else.

Correct Alignment

After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors.

Installing ZFS

deb http://www.coker.com.au wheezy zfs

I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn’s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik’s binary packages but I think it’s best to rebuild Ubuntu packages to use on Debian.

The server in question hasn’t gone live in production yet (it turns out that we don’t have agreement on what the server will do). But so far it seems to be working OK.

The Retina Display

Last night I played with an Apple Macbook Pro with the new Retina Display (Wikipedia link). Wikipedia cites some controversy about whether the display actually has higher resolution than the human eye can perceive. When wearing glasses my vision is considerably better than average (I have average vision without glasses) and while kneeling in front of the Macbook I couldn’t easily distinguish pixels. So Apple’s marketing claims seem technically correct to me.

When I tested the Macbook Pro I found that the quality of the text display was very high, even now the 1680*1050 display on my Thinkpad T61 looks completely crap when compared to the 2880*1800 display on the Macbook. The Macbook was really great for text and for a JPEG that was installed on the system for testing. But unfortunately pictures on web sites didn’t look particularly good. Pictures on my blog looked quite poor and pictures returned by a Google search for “art” didn’t look that great either. I wonder if Safari (the Apple web browser) isn’t properly optimised for the display or if there is something that we should do when preparing pictures for web sites to make them look better on Safari.

The retina display has a 71% greater DPI which means 2.93* the total number of pixels of my Thinkpad. The overall quality of the experience for me (apart from web pictures) seems more like a factor of 2.93 when compared with my Thinkpad than a factor of 1.71. This has to be one of the most desirable products I’ve seen from a company that’s opposed to freedom for it’s users. I’m not about to buy one though, $2,300 is a lot of money for a system that can’t be upgraded, repaired, or recycled, and doesn’t even have an Ethernet port. I’m sure that if I bought one I would discover that it some of the hardware features don’t work properly with Linux.

The new Apple design trend of making it impossible to repair anything works reasonably well for phones and tablets which are cheap enough that they are hardly worth repairing when they have been used for a while. Lots of people can afford to spend about $600 on something that may be discarded after a year or two, but very few people can afford to spend more than $2,000 on such a disposable product.

Why is Apple the only company producing systems with such displays? If someone produced regular PCs that have the expected features (including an Ethernet port) with such a display at a lower price then I’m sure that there would be a great demand.

Links July 2012

The New York Times has an interesting article about “hacker hostels” [1]. I had an idea for similar things after watching a Japanese movie about Tokiwa-sÅ – a shared apartment for Manga artists which among others inspired the creator of Astro Boy [2].

The TED blog has an interesting interview with William Noel about open access to art and historical data [3]. Most of his talk concerns an Archimedes codex which has been recovered and published on the Internet. He advocates publishing all manner of art and historical data under a Creative Commons license.

The education system is often criticised for trying too hard to make children feel successful and not teaching them skills needed to be successful, it seems that the US military fails the same way in it’s war games [4].

Webroot.com published an interesting article last year about the first BIOS rootkit in the wild [5]. I really wish that they would design motherboards with a switch to enable BIOS writing which would default to “off”. I recently did a poll at a LUG meeting and found that only half the audience had updated the BIOS on most systems they owned, if the most technical people generally don’t need a dangerous feature then it should probably be disabled by default.

Matthew Wright wrote an interesting article about the costs of upgrading the electricity grid in Australia vs the costs of upgrading air-conditioners [6]. It seems that it would be a lot cheaper for the government to buy everyone a new air-conditioner than to upgrade the grid.

Owl City has a post of 10 Myths About Introverts [7]. That could probably be titled 10 Myths About Aspies and still be correct.

Susan Cain gave the most popular talk of TED 2012 about “The Power of Introverts”, here is an interesting interview about the talk and Introversion [8].

Some Proprietary Platform Issues

Android vs iPad

I’m currently in discussions with a client about a potential future project which involves a tablet computer talking to some electronic equipment. The options are an Android tablet and an iPad. One advantage of Android is that it runs on devices of all shapes and sizes, so we can choose a device that fits the need rather than designing everything around the iPad.

But the real problem with iPad is Apple. To run an app on an iPad you need to submit it to Apple, hopefully get it accepted into the App Market, then install it. This process causes some delay, a minor fee, and has the potential to derail the project if Apple doesn’t accept the app on the first try. With Android there is no need to even deal with Google, the app can be installed directly without the Google Play store.

I may end up working with an iPad (which admittedly is really nice hardware), but it seems most likely that the project in question will run on Android only.

Windows vs Linux and Apple OS/X

One of my clients recently paid a web development company to redevelop his web site. I turned out that the web developers in question only knew how to develop for Windows and my client didn’t discover this until too late. Now a site that’s currently using a small fraction of the resources on a $80 per month Linode instance will run on a Windows virtual server costing $300 per month (which includes SQL server license).

The Windows virtual server will probably be managed (because my client uses only Apple and Linux systems and doesn’t employ anyone with Windows skills) which adds an extra $100 per month. If the server isn’t managed then they will have to hire someone to apply patches and that won’t necessarily be cheaper.

So using Windows is going to cost my client an extra $400 per month when compared to the possibility of running a Linux system on the existing virtual server. Even if my client had someone with Windows skills to run the server it would still be an extra $300 per month. If the NBN was available then my client could run a Windows server in their office, but it’s not yet available in their area.

Even for a company that employs people with more Windows skills than Linux skills there are still economic factors in favor of Linux due to smaller hardware requirements and the lack of license fees for all the core software (OS, database server, web server, etc).

Summary

These anecdotes aren’t unusual, it’s the sort of thing that happens all the time. Sometimes the result is good (EG avoiding the iPad), sometimes it isn’t (being stuck with a proprietary web service).

I think I’ll have to suggest to my clients that every contract have a “no proprietary software” clause. Contracts can be amended if there is a reason, but it seems best to make a preemptive strike against companies that sneak proprietary software in and cause significant unexpected expense and difficulty.

Breaking SATA Connectors

I’ve just broken my second SATA connector. This isn’t a lot considering the number of hard drives I’ve worked with, but it’s still really annoying as I generally don’t break things.

The problem is that unplugging a SATA cable requires pushing a little clip, this isn’t overly difficult but it unfortunately doesn’t fit well with habits formed from previous hardware. The power cables used for hard drives based on the ST-506 interface which was copied for the IDE interface was large and had a fairly tight fit. Removing such a cable requires a significant amount of force – which is about the same as the amount of force required to break a SATA connector.

When I first started using PCs a reasonably configured AT system cost over $5,000 (maybe something like $10,000 in today’s money). With that sort of price hardly anyone had a set of test PCs. When hardware prices dropped such that hard drives of reasonable size became reasonably affordable on the second-hand market I bought more disks and used some for extra storage and some for testing software. As there was nothing like VMWare for testing OS images the way to test a new OS was to plug in a different hard drive and boot it. So I got a lot of practice at removing IDE power cables with as much force as was necessary.

Now I own a pile of test PCs, SATA disks less than 100G are free, I use Xen for a lot of my testing, and generally I have much less need to swap hard drives around. In most situations in which I would swap hard drives in the 90’s I will now swap PCs and I have piles of PCs ready for this purpose. So I haven’t had enough practice with SATA disks to develop habits for safely removing them.

So far this lack of habit development has resulted in damaging two disks due to changing drives while not concentrating enough. Fortunately duct-tape works well for holding a SATA connector in place when the plastic that attaches to the clip is broken.

Postfwd and Local Only Email

Over a year ago when I was considering my first Android phone purchase I setup a test account on my mail server so that I could test email clients on phones and tablets. I used a short password because I didn’t want to type a lot on small screens and because typing a password into a random system owned by someone else isn’t particularly secure anyway. Then I forgot about the account until I noticed that my mail server was sending out spam.

Next time I setup such a test account I’ll put rules similar to the following in my Postfwd [1] configuration to stop Postfix from sending such messages. That will prevent the test account from receiving mail from outside or sending mail out of the server. The former is optional (getting a few thousand spam messages in an unused test account is no big deal) but the latter is needed to prevent getting my server blacklisted.

id=R_test_recipient ; recipient==test@coker.com.au ; sender!~.*@coker.com.au ; action=REJECT
id=R_test_sender ; sender==test@coker.com.au ; recipient!~.*@coker.com.au ; action=REJECT

[1] http://postfwd.org/

Links June 2012

This Youtube video is an amusing satire of EULAs and copyright law as applied to uploading consciousness [1].

Washington’s Blog has an insightful article about the way that the lack of trust in the US is killing their economy [2]. It seems that as long as the 1% are allowed to get away with breaking the law the US economy won’t recover. It’s something we should all keep in mind at election time, let’s not be like the US.

AirBnB is an interesting service to allow people to rent a room or an apartment, a quick scan indicates that it’s a lot cheaper than hotels and offers many good locations [3]. It’s probably of most interest to the more social people though which is a down side for me.

Queensland’s highest court has ruled that “vilification of homosexuals is also vilification of bisexuals” because “an essential aspect of bisexuality is a sexual feeling of a person of the same sex, that is, homosexuality” [4]. Anyone who didn’t find that totally obvious could simply consult any dictionary or encyclopedia to find out. But the Australian legal system needed a 46 page ruling. We really need some sanity in the courts.

Father Gregory Boyle founded an organisation named Homeboy Industries with the purpose of providing jobs for people with criminal records [5]. It’s amazing the way he is helping people turn their lives around and it’s apparently a lot cheaper than sending them to jail.

Targeted Advertising

Don Marti has written another blog post about targeted advertising [1]. His main point is that when a company uses the most targeted adverts (such as Google advertising) everyone knows that they are paying a small number of cents per click and nothing for the people who don’t click. This compares to TV adverts which cost a lot of money and for which most viewers either leave the room or use fast-forward. Therefore using Google adverts doesn’t send a signal about the amount of money invested in the products. Don also cited an example of a company sponsoring an OK Go film clip, that was a great idea, it shows that the company can do expensive things which are also a bit creative and fans will thank them (watch all the OK Go videos on Youtube, they are great).

The next question is how else companies can advertise? One thing I’d really like to see is sponsorship of authors. Pick an author and pay them a salary with paid editorial services for releasing a book a year for free in HTML and ebook formats. Having a fixed salary is a significant benefit when it comes time to apply for a mortgage or plan a holiday and being able to freely distribute books would be a significant benefit for an author who hasn’t got a large fan base.

In the computer industry it seems that there’s a lot of potential for sponsoring people who produce free things. That ranges from free software and designs for free hardware to blog posts and documentation. Five years ago Sun had a blogging contest and my friend Dave Hall won a server that was worth $21K [2]. It would be nice if some other companies started doing similar things and if Sun did a repeat so some other people I like could get some free kit.

New SE Linux Policy for Wheezy

I’ve just uploaded a new SE Linux policy for Debian/Wheezy. It now works correctly with systemd and Chromium, two significant features that I wanted for Wheezy. Now it turns out that we have until the end of the month for Wheezy updates, so I may get another version of the policy uploaded before then. If so it will only be for relatively minor changes, I think that most SE Linux users would be reasonably happy with policy the way it is. Anything that doesn’t work now can probably be solved by local configuration changes.

execmem

The current version of KDE in Debian is 4.8.4, it seems that large parts of the KDE environment depend on execmem access, this includes kwin and plasma-desktop. Basically there is no possibility of having a KDE desktop environment without those programs and therefore KDE depends on execmem access.

Debugging this is difficult as the important programs SEGV when denied execmem access and the KDE crash handler really gets in the way of debugging it – running /usr/bin/plasma-desktop results in the process forking a child and detaching from the gdb session.

The most clear example of an execmem issue in KDE is from the program /usr/lib/kde4/libexec/kwin_opengl_test which gives the following error:
LLVM ERROR: Allocation failed when allocating new memory in the JIT
Can’t allocate RWX Memory: Permission denied

To make this work you run the command “setsebool -P allow_execmem 1” which gives many domains the ability to create writable-executable memory regions.

I raised this issue for discussion on the SE Linux mailing list and Hinnerk van Bruinehsen wrote an informative message in response summarising the situation [1]. It seems that it’s possible to compile some of the programs in question to not use the JIT and therefore not require such access and there is a build option in Gentoo to allow it. But it’s impractically difficult for me to fork KDE in Debian so the only option is to recommend that people enable the allow_execmem boolean for Debian desktop systems running SE Linux.

[1] http://marc.info/?l=selinux&m=134011618909818&w=2

SASL Authentication and Debian/Wheezy

After upgrading a mail server to Debian/Unstable (which will soon be released as Wheezy) I started getting SASL errors.

535 5.7.8 Error: authentication failed: no mechanism available

The SMTP protocol gave the above error for both LOGIN and PLAIN methods.

SASL LOGIN authentication failed: no mechanism available

The postfix/smtpd process logged messages like the above in syslog.

It turned out that the “auxprop_plugin: mysql” line had to be removed and replaced with the following two lines due to a change in the way SQL plugins are managed:

auxprop_plugin: sql
sql_engine: mysql

Also the SQL query needed to have “%u” replaced with “%u@%r” because we now have user and realm provided separately.

etbe – Russell Coker

Archives

Categories

ZFS on Debian/Wheezy

BTRFS vs ZFS

ZFS on Wheezy

Root on ZFS

Correct Alignment

Installing ZFS

The Retina Display

Links July 2012

Some Proprietary Platform Issues

Android vs iPad

Windows vs Linux and Apple OS/X

Summary

Breaking SATA Connectors

Postfwd and Local Only Email

Links June 2012

Targeted Advertising

New SE Linux Policy for Wheezy

execmem

SASL Authentication and Debian/Wheezy

Archives

Email and RSS

Archives

Categories

Tags

BTRFS vs ZFS

ZFS on Wheezy

Root on ZFS

Correct Alignment

Installing ZFS

Android vs iPad

Windows vs Linux and Apple OS/X

Summary

execmem

Archives

Email and RSS