Is Portslave Still Useful?

Portslave is a project that was started in the 90’s to listen to a serial port and launch a PPP or SLIP session after a user has been authenticated, I describe it as a “project” not a “program” because a large part of it’s operation is via a shared object that hooks into pppd, so if you connect to a Portslave terminal server and just start sending PPP data then the pppd will be launched and use the Portslave shared object for authentication. This dual mode of operation makes it a little tricky to develop and maintain, every significant update to pppd requires that Portslave be recompiled at the minimum, and sometimes code changes in Portslave have been required to match changes in pppd. CHAP authentication was broken in a pppd update in 2004 and I never fixed it, as an aside the last significant code change I made was to disable CHAP support, so I haven’t been actively working on it for 9 years.

I took over the Portslave project in 2000, at the time there were three separate forks of the project with different version numbering schemes. I used the release date as the only version number for my Portslave releases so that it would be easy for users to determine which version was the latest. Getting the latest version was very important given the ties to pppd.

When I started maintaining Portslave I had a couple of clients that maintained banks of modems for ISP service and for their staff to connect to the Internet. Also multi-port serial devices were quite common and modems where the standard way of connecting to the Internet.

Since that time all my clients have ceased running modems. Most people connect to the Internet via ADSL or Cable, and when people travel they use 3G net access via their phone which is usually cheaper, faster, and more convenient than using a modem. The last code changes I made to Portslave were in 2010, since then I’ve made one upload to Debian for the sole purpose of compiling against a new version of pppd.

I have no real interest in maintaining Portslave, it’s no longer a fun project for me, I don’t have enough spare time for such things, and no-one is paying me to work on it.

Currently Portslave has two Debian bugs, one is from a CMU project to scan programs for crashes that might indicate security flaws, it seems that Portslave crashes if standard input isn’t a terminal device [1]. That one shouldn’t be difficult to solve.

The other Debian bug is due to Portslave being compiled against an obsolete RADIUS client library [2]. It also shouldn’t be that difficult to fix, when I made it use libradius1 that wasn’t a difficult task and it should be even easier to convert from one RADIUS library to another.

But the question is whether it’s worth bothering. Is anyone using Portslave? Is anyone prepared to maintain it in Debian? Should I just file a bug report requesting that Portslave be removed from Debian?

Hetzner now Offers SSD

Hetzner is offering new servers with SSD, good news for people who want to run ZFS (for ZIL and/or L2ARC). See the EX server configuration list for more information [1]. Unfortunately they don’t specify what brand of SSD, this is a concern for me as some of the reports about SSD haven’t been that positive, getting whichever SSD is cheapest isn’t appealing. A cheap SSD might be OK for L2ARC (read cache), but for ZIL (write cache) reliability is fairly important. If anyone has access to a Hetzner server with SSD then please paste the relevant output of lsscsi into a comment.

The next issue is that they only officially offer it on the new “EX 8S” server. SSD will be of most interest to people who also want lots of RAM (the zfsonlinux.org code has given me kernel panics when running with a mere 4G of RAM – even when I did the recommended tuning to reduce ARC size). Also people who want more capable storage options will tend to want more RAM if only for disk caching.

But I’m sure that there are plenty of people who would be happy to have SSD on a smaller and cheaper server. The biggest SSD offering of 240G is bigger than a lot of servers. I run a Hetzner server that has only 183G of disk space in use (and another 200G of backups). If the backups were on another site then the server in question could have just a RAID-1 of SSD for all it’s storage. In this case it wouldn’t be worth doing as the server doesn’t have much disk IO load, but it would be nice to have the option – the exact same server plus some more IO load would make SSD the ideal choice.

The biggest problem is that the EX 8S server is really expensive. Hard drives which are included in the base price for cheaper options are now expensive additions. A server with 2*3TB disks and 2*240G SSD is E167 per month! That’s more expensive than three smaller servers that have 2*3TB disks! The good news for someone who wants SSD is that the Hetzner server “auction” has some better deals [2]. As is always the case with auction sites the exact offers will change by the moment, but currently they offer a server with 2*120G SSD and 24G of RAM for E88 per month and a server with 2*120G SSD, 2*1.5T HDD, and 24G of RAM for E118. E88 is a great deal if your storage fits in 240G and E118 could be pretty good if you only have 1.5T of data that needs ZFS features.

The main SSD offering is still a good option for some cases. A project that I did a couple of years ago would probably have worked really well on a E167/month server with 2*3TB and 2*240G SSD. It was designed around multiple database servers sharding the load which was largely writes, so SSD would have allowed a significant reduction in the number of servers.

They also don’t offer SSD on their “storage servers” which is a significant omission. I presume that they will fix that soon enough. 13 disks and 2 SSD will often be more useful than 15 disks. That’s assuming the SSD doesn’t suck of course.

The reason this is newsworthy is that most hosted server offerings have very poor disk IO and no good options for expanding it. For servers that you host yourself it’s not too difficult to buy extra trays of disks or even a single rack-mount server that has any number of internal disks in the range 2 to 24 and any choice as to how you populate them. But with rented servers it’s typically 2 disks with no options to add SSD or other performance enhancements and no possibility of connecting a SAN. As an aside it would still be nice if someone ran a data center that supported NetApp devices and gave the option of connecting an arbitrary number of servers to a NetApp Filer (or a redundant pair of Filers). If anyone knows of a hosting company that provides options for good disk IO which are better than just providing SSD or cheaper than E167 per month then please provide the URL in a comment.

Update: It seems that I can get SSD added to one of the cheaper servers. This is a good option as I have some servers that already have the “flexi-pack” due to a need for more IP addresses.

ZFS on Debian/Wheezy

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable.

BTRFS vs ZFS

Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don’t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don’t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity.

I am about to install a system to be a file server and Xen server which probably isn’t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won’t be upgraded a lot I’ll start with Debian/Wheezy even though it isn’t released yet because the system will be in use without much change well after Squeeze security updates end.

ZFS on Wheezy

Getting ZFS to basically work isn’t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn’t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn’t particularly difficult but it’s a pity that the zfsonlinux site didn’t support recent kernels.

Root on ZFS

The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device.

Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn’t ideal, but when you have an OS image that is backed up (and not so important) and a data store that’s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it.

There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort.

The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn’t work with the Debian initramfs for some reason and modules failed to load. This wasn’t necessarily a show-stopper as I can modify such things myself, but it’s another painful thing to manage and another way that the system can potentially break on upgrade.

The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the “zfs set mountpoint” command. This of course means that it doesn’t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn’t a good option. So for this new server I’ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else.

Correct Alignment

After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors.

Installing ZFS

deb http://www.coker.com.au wheezy zfs

I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn’s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik’s binary packages but I think it’s best to rebuild Ubuntu packages to use on Debian.

The server in question hasn’t gone live in production yet (it turns out that we don’t have agreement on what the server will do). But so far it seems to be working OK.

New Version of Memlockd

I’ve just released a new version of Memlockd, a daemon to lock essential files in RAM to increase the probability of recovering a system that is paging excessively [1].

The new features are:
Using Debian/Wheezy paths for shared objects on i386 and amd64.

Added a new config file option to not log file not found errors so we don’t see i386 errors on amd64 and amd64 errors on i386.

Added a systemd service file which I haven’t yet tested, but I won’t get to test it for a while so for the moment I’ve released it and hope that the person who submitted the file got it right and that my minor change didn’t break it.

Added a run-parts style config directory, default is /etc/memlock.d and now the config file uses a % to chain to another file or directory.

So I fixed all but one of the Debian bugs in time for Wheezy, provided that the systemd stuff works. If someone has time to test it with systemd for me then that would be great!

Another USB Flash Failure

I previously wrote about a failure of a USB flash device in my Internet gateway [1]. I have since had another failure in the same system, so both the original 4G devices are now dead. That’s two dead devices in 10 weeks. It could be that the USB devices that I got for free at an exhibition were just really cheap, I’m sure that they weren’t expecting them to be used in that way. The devices from the same batch which are used for their intended purpose (sneaker-net file sharing) are still working well. But in any case I’m not going to resume this experiment until warmer weather. At this time of year some extra heat dissipation from computer gear in my home is more like a feature and less like a bug.

The second USB device to fail appeared to have it’s failure in the Ext4 journal (the errors were reported at around sector 2000), I didn’t keep a record of the problem with the first device, but from memory I think it was much the same.

Rumor has it that cheap flash storage devices don’t implement wear-levelling to avoid patent infringement. If that rumor is correct then any filesystem that uses a fixed journal in the same way as Ext3/4 is probably unsuitable for any serious use on such devices, while a filesystem based on Copy On Write will probably perform better. In Spring I’ll try using BTRFS on cheap USB flash devices and see if that works better. I have another spare device from the same batch to test so I can eliminate hardware differences. I can’t do enough tests to be a good statistical sample, but if a device lasts from Spring to Autumn using BTRFS with the same use that caused failures with Ext4 in a few weeks then I will consider it a strong indication that BTRFS is better than Ext3/4 for such uses.

For the next 5 months or so I’ll be using a hard drive in my Internet gateway system again.

Flash Storage Update

Last month I wrote about using USB flash storage devices for my firewall and Squid proxy [1]. 4 days ago it failed, the USB device used for the root filesystem stopped accepting write requests. The USB device which is used for /var/spool/squid is still going well after almost 5 months of intensive use while the USB device for the root filesystem failed after 24 days of light use. Both USB devices were of the same model and were obtained at the same time. Presumably one of them was just defective.

I’m now using an old 1G USB device for the root filesystem. When using it on less ancient systems with USB 2.0 there was an obvious speed difference between the 1GB and 4GB devices. But when run on USB 1.2 they can both support the maximum speed of the port so performance probably isn’t any worse. Not that it really matters for the root filesystem, the server is supposed to run without a break for long periods of time so if boot time becomes a performance issue then whatever is causing the reboots will be a much bigger problem.

It’s annoying to have a device fail and the failure rate for USB flash devices running 24*7 is looking rather bad at the moment. But I’m confident that things will run well from now on.

USB Flash Storage

For some years I have had my Internet gateway/firewall system in a cupboard in my bedroom. While I don’t mind some computer noise (I’ve slept near a server for most of the last 22 years) it’s good to have it as quiet as possible so getting rid of the hard drive is desirable.

I considered buying an IDE flash drive, but I’d like to continue my trend of not paying for hardware so I chose to use some USB flash devices that HP was giving away at a seminar (thanks HP – as an aside the P3 system is an old Compaq Desktop system). So I’ve got one 4G USB device for root and one for Squid.

For the past few months I’ve had /var/spool/squid be a USB flash device. I considered using RAID-0 for that filesystem because the computer is a P3 and only has USB 1.2 and thus a maximum theoretical transfer rate of 1.5MB/s and a maximum real-world rate of about 1MB/s. But my ADSL connection doesn’t seem able to sustain much more than 1MB/s and Squid doesn’t write data synchronously so in all my tests the USB speed hasn’t affected HTTP performance.

One issue that has delayed my move to all USB is the difficulty of booting as the P3 system in question doesn’t support booting from USB. I considered creating a boot CD that loads the kernel from the USB device, but that seemed a little painful and also relies on the CD-ROM drive working – which isn’t a great idea for a system that runs 24*7 in a dusty cupboard. I ended up using GRUB on the IDE hard drive to load the kernel and initrd and then mount a USB device as root, this seems to work and the command “hdparm -S6 /dev/sda” in /etc/rc.local makes the hard drive go to sleep once the system is booted.

The only technical parts of the process were putting in the UUIDs of the filesystems in /etc/fstab (because I can’t be sure which USB device will be found first) and creating a new initramfs with modules for USB storage listed in /etc/initramfs-tools/modules so that a USB device could be the root filesystem.

The firewall system is now a bit quieter and based on my last tests of hard drive power use will probably dissipate about 5-7W less heat. The next thing to do is wait and see if it keeps running or falls over. ;)

Cooling a Thinkpad

Late last year I wrote about the way that modern laptops suck [1]. One of the problems that inspired that post was the excessive heat generated by my Thinkpad T61.

There is a partial solution to this, Fool Control explains how the kernel option pcie_aspm=force can be used on kernels from 2.6.38 onwards to solve a heat regression problem [2]. I applied this to my Thinkpad T61 and the result was that on a cool evening (ambient temperature about 24C) the temperature changed from 85C to 66C on the NVidia video card, and for the “virtual devices” it changed from 80C and 78C to 60C and 61C. I’m not sure exactly what each of those measurements refers to, but it seems that the change was somewhere between 17C and 20C.

This changes the system from being almost unbearable to use to being merely annoyingly warm.

I’m not going to make my laptop be my primary computing device again though, the combination of a desktop system with a 27″ monitor and an Android phone is working quite well for me [3]. But I haven’t yet got version control systems working for all my software. Also Wouter suggested using NBD which is something I haven’t got working yet and probably won’t until I can swap on it and therefore have a diskless workstation. Finally I still haven’t got the “Chrome to Phone” browser extension working such that a page I’m viewing at home can be loaded on my phone.

Magic entries for BTRFS and Software RAID

I’ve just discovered that the magic database for the file(1) command in Debian/Unstable has no support for Linux Software RAID and that it’s support for BTRFS is lacking (no reporting of space used, number of devices, or the UUID). Below is my first draft of a change to fix these problems. I would appreciate it if someone with a big-endian system could test these out and let me know how they go, I suspect that I will have to change the “lelong” types to “long” but I’m not sure.

4096 lelong 0xa92b4efc Linux Software RAID
>4100 lelong x version 1.2 (%d)
>4112 belong x UUID=%8x:
>4116 belong x \b%8x:
>4120 belong x \b%8x:
>4124 belong x \b%8x
>4128 string x name=%s
>4168 lelong x level=%d
>4188 lelong x disks=%d

0 lelong 0xa92b4efc Linux Software RAID
>4 lelong x version 1.1 (%d)
>16 belong x UUID=%8x:
>20 belong x \b%8x:
>24 belong x \b%8x:
>28 belong x \b%8x
>32 string x name=%s
>72 lelong x level=%d
>92 lelong x disks=%d

# BTRFS
0x10040 string _BHRfS_M BTRFS Filesystem
>0x1012b string >\0 label "%s",
>0x10090 lelong x sectorsize %d,
>0x10094 lelong x nodesize %d,
>0x10098 lelong x leafsize %d,
>0x10020 belong x UUID=%8x-
>0x10024 beshort x \b%4x-
>0x10026 beshort x \b%4x-
>0x10028 beshort x \b%4x-
>0x1002a beshort x \b%4x
>0x1002c belong x \b%8x,
>0x10078 lequad x %lld/
>0x10070 lequad x \b%lld bytes used,
>0x10088 lequad x %lld devices

How to Start Learning Linux

I was asked for advice on how to start learning Linux. Rather than replying via email I’m writing a blog post for future people who ask such questions and also to get comments from other people which may provide information I missed.

Join a LUG

The best thing to do is to start by joining your local Linux Users Group (LUG). Linux International maintains a list of LUGs that is reasonably comprehensive [1]. Even if there isn’t a LUG near enough for you to attend meetings you can learn a lot from a mailing list of a LUG that’s close to your region. There is usually no great reason not to join the mailing list of a LUG in a different region or country, but a local LUG is that the advice will often be tailored to issues such as the local prices of hardware and the practices of your government.

Also note that Linux International doesn’t list all LUGs, the MLUG group in Melbourne [2] and the BLUG group in Ballarat [3] aren’t listed. Anyone who joins LUG (the group based in Melbourne, Victoria that I’m a member of) will be advised of the smaller groups in the region if they ask on the list.

As an aside it would probably make sense for the main LUV web page [4] to have links to local LUGs and to the LI page of users’ groups and for other LUGs to do the same. It’s pretty common for a Google search to turn up the web site of a LUG that’s near the ideal location but not quite right. Also it would be good if LUV could have a link to the Victorian Linux Users Group in Canada – this should reduce the confusion a bit and they have a link to us [5].

Play with Linux

Get a spare PC (with no important data) and try installing different distributions of Linux on it. Make sure that it never has anything particularly important so you can freely try things out without worrying about the risk of losing data. Part of the learning process usually involves breaking a system so badly that it needs to be reinstalled. Linux can run on really old hardware, an old system with 64M of RAM will do for learning (but 128M will really be preferred and 256M will be even better).

Learn with other Beginners

LUV has a very active beginners group, with a beginners mailing list and special beginners meetings. A group that has such things will be more helpful as you can easily learn from other people who are at a similar level to you. Also you can spend time learning Linux with friends, just spend a weekend with some friends who want to learn Linux and play with things – you can often learn more by trying things than by reading books etc.

Do some Programming

One of the advantages of Linux (and other Free Software OSs) is that it comes with a full range of programming languages for free. You can get a much greater understanding of an OS by writing programs for it and a typical Linux distribution gives you all the tools you need.

Any other Ideas?

Does anyone have any other suggestions? Please leave a comment.