wp-spamshield

Yesterday I installed the wp-spamshield plugin for WordPress [1]. It blocks automated comment spam systems by using JavaScript and cookies, apparently most spammers can’t handle that. Before I installed it I was getting hundreds of spam comments per day even with the block spam by math plugin enabled. Now I’ve had it running for 24 hours without any spam. The real advantage of this is that now when a legitimate comment gets flagged as spam I’ll notice it, previously I was deleting hundreds or thousands of comments at a time without reading them.

deb http://www.coker.com.au wheezy wordpress

The above repository has the wordpress-wp-spamshield package for Debian/Wheezy. I have no immediate plans for uploading it to Debian because the security support for WordPress plugins doesn’t fit in with the Debian model. I am prepared to negotiate about this if someone has good reasons for including it or any of the other WordPress plugins I’ve packages.

My packaging work is under the GPL (of course) so any DD who disagrees with me could just rebuild the package and upload it. Within Debian there is no rule taking another DD’s GPL’d code that they decided not to upload and then uploading it. There is a consensus that such things are not appropriate without permission, but anyone who wishes can take this blog post as permission.

Fixing Strange Directory Write Access

type=AVC msg=audit(1403622580.061:96): avc:  denied  { write } for  pid=1331 comm="mysqld_safe" name="/" dev="dm-0" ino=256 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=dir
type=SYSCALL msg=audit(1403622580.061:96): arch=c000003e syscall=269 success=yes exit=0 a0=ffffffffffffff9c a1=7f5e09bfe798 a2=2 a3=2 items=0 ppid=1109 pid=1331 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="mysqld_safe" exe="/bin/dash" subj=system_u:system_r:mysqld_safe_t:s0 key=(null)

For a long time (probably years) I’ve been seeing messages like the above in the log from auditd (/var/log/audit/audit.log) when starting mysqld. I haven’t fixed it because the amount of work exceeded the benefit, it’s just a couple of lines logged at every system boot. But today I decided to fix it.

The first step was to find out what was going on, I ran a test system in permissive mode and noticed that there were no attempts to create a file (that would have been easy to fix). Then I needed to discover which system call was triggering this. The syscall number is 269, the file linux/x86_64/syscallent.h in the strace source shows that 269 is the system call faccessat. faccessat(2) and access(2) are annoying cases, they do all the permission checks for access but don’t involve doing the operation so when a program uses those system calls but for some reason doesn’t perform the operation in question (in this case writing to the root directory) then we just get a log entry but nothing happening to examine.

A quick look at the shell script didn’t make the problem obvious, note that this is probably obvious to people who are more skilled at shell scripting than me – but it’s probably good for me to describe how to solve these problems every step of the way. So the next step was to use gdb. Here is the start of my gdb session:

# gdb /bin/sh
[skipped]
Reading symbols from /bin/dash…(no debugging symbols found)…done.
(gdb) b faccessat
Breakpoint 1 at 0x3960
(gdb) r -x /usr/bin/mysqld_safe
[lots skipped]
+ test -r /usr/my.cnf
Breakpoint 1, 0x00007ffff7b0c7e0 in faccessat ()
from /lib/x86_64-linux-gnu/libc.so.6

After running gdb on /bin/sh (which is a symlink to /bin/dash) I used the “b” command to set a breakpoint on the function faccessat (which is a library call from glibc that calls the system call sys_faccessat()). A breakpoint means that program execution will stop when the function is called. I run the shell script with “-x” as the first parameter to instruct the shell to show me the shell commands that are run so I can match shell commands to system calls. The above output shows the first call to faccessat() which isn’t interesting (it’s testing for read access).

I then ran the “c” command in gdb to continue execution and did so a few times until I found something interesting.

+ test -w / -o root = root
Breakpoint 1, 0x00007ffff7b0c7e0 in faccessat ()
from /lib/x86_64-linux-gnu/libc.so.6

Above is the interesting part of the gdb output. It shows that the offending shell command is “test -w /“.

I filed Debian bug #752593 [1] with a patch to fix this problem.

I also filed a wishlist bug against strace asking for an easier way to discover the name of a syscall [2].

Is Portslave Still Useful?

Portslave is a project that was started in the 90’s to listen to a serial port and launch a PPP or SLIP session after a user has been authenticated, I describe it as a “project” not a “program” because a large part of it’s operation is via a shared object that hooks into pppd, so if you connect to a Portslave terminal server and just start sending PPP data then the pppd will be launched and use the Portslave shared object for authentication. This dual mode of operation makes it a little tricky to develop and maintain, every significant update to pppd requires that Portslave be recompiled at the minimum, and sometimes code changes in Portslave have been required to match changes in pppd. CHAP authentication was broken in a pppd update in 2004 and I never fixed it, as an aside the last significant code change I made was to disable CHAP support, so I haven’t been actively working on it for 9 years.

I took over the Portslave project in 2000, at the time there were three separate forks of the project with different version numbering schemes. I used the release date as the only version number for my Portslave releases so that it would be easy for users to determine which version was the latest. Getting the latest version was very important given the ties to pppd.

When I started maintaining Portslave I had a couple of clients that maintained banks of modems for ISP service and for their staff to connect to the Internet. Also multi-port serial devices were quite common and modems where the standard way of connecting to the Internet.

Since that time all my clients have ceased running modems. Most people connect to the Internet via ADSL or Cable, and when people travel they use 3G net access via their phone which is usually cheaper, faster, and more convenient than using a modem. The last code changes I made to Portslave were in 2010, since then I’ve made one upload to Debian for the sole purpose of compiling against a new version of pppd.

I have no real interest in maintaining Portslave, it’s no longer a fun project for me, I don’t have enough spare time for such things, and no-one is paying me to work on it.

Currently Portslave has two Debian bugs, one is from a CMU project to scan programs for crashes that might indicate security flaws, it seems that Portslave crashes if standard input isn’t a terminal device [1]. That one shouldn’t be difficult to solve.

The other Debian bug is due to Portslave being compiled against an obsolete RADIUS client library [2]. It also shouldn’t be that difficult to fix, when I made it use libradius1 that wasn’t a difficult task and it should be even easier to convert from one RADIUS library to another.

But the question is whether it’s worth bothering. Is anyone using Portslave? Is anyone prepared to maintain it in Debian? Should I just file a bug report requesting that Portslave be removed from Debian?

SE Linux Things To Do

At the end of my talk on Monday about the status of SE Linux [1] I described some of the things that I want to do with SE Linux in Debian (and general SE Linux stuff). Here is a brief summary of some of them:

One thing I’ve wanted to do for years is to get X Access Controls working in Debian. This means that two X applications could have windows on the same desktop but be unable to communicate with each other by any of the X methods (this includes screen capture and clipboard). It seems that the Fedora people are moving to sandbox processes with Xephyr for X access (see Dan Walsh’s blog post about sandbox -X [2]). But XAce will take a lot of work and time is always an issue.

An ongoing problem with SE Linux (and most security systems) is the difficulty in running applications with minimum privilege. One example of this is utility programs which can be run by multiple programs, if a utility is usually run by a process that is privileged then we probably won’t notice that it requires excess privileges until it’s run in a different context. This is a particular problem when trying to restrict programs that may be run as part of a user session. A common example is programs that open files read-write when they only need to read them, if the program then aborts when it can’t open the file in question then we will have a problem when it’s run from a context that doesn’t grant it write access. To deal with such latent problems I am considering ways of analysing the operation of systems to try and determine which programs request more access than they really need.

During my talk I discussed the possibility of using a shared object to log file open/read/write to find such latent problems. A member of the audience suggested static code analysis which seems useful for some languages but doesn’t seem likely to cover all necessary languages. Of course the benefit of static code analysis is that it will catch operations that the program doesn’t perform in a test environment – error handling is one particularly important corner case in this regard.

Creating WordPress Packages

deb http://www.coker.com.au wheezy wordpress

I maintain Debian packages of a number of WordPress themes and plugins for my personal use which I am not planning to upload to Debian due to the maintenance and security issues. Generally the way things work with WordPress packages (and apparently most things in PHP) is that new versions are released whenever the author feels like it with little documentation and often now way of determining whether it’s a security issue. When there is a security issue it’s often fixed in a new version that includes new features giving no good option for someone who was happy with the old functionality and just wants a secure system. This isn’t the way we like to do things in Debian.

The result of this is that I maintain a number of packages for my personal use (and for the benefit of any interested people on the Internet) that often get new updates. I’ve written the below script to create a new version of a Debian package. It searches my repository for the most recent .debian.tar.gz file for the package, applies that, runs dch -i to update the changelog, and then builds the package. So far this has only been tested on one package, I expect that I’ll have to put a sed command in there to cover the case where the zip file name doesn’t match what I want as the package name and I’ll probably find other bugs in future, but I think it’s good enough to publish now.

#!/bin/bash
set -e
REPOSITORY=/home/whatever
unzip $1
FILE=$(basename $1)
PACKAGE=$(echo $FILE | sed -e "s/\..*$//")
LEN=$(($(echo $PACKAGE | wc -c)+1))
VER=$(echo $FILE | cut -c ${LEN}-200 | sed -e s/.zip//)
DIRNAME=wordpress-${PACKAGE}-${VER}
mv $PACKAGE $DIRNAME
tar czf wordpress-${PACKAGE}_${VER}.orig.tar.gz ${DIRNAME}
cd $DIRNAME
tar xzf $(ls -tr ${REPOSITORY}/wordpress-${PACKAGE}_*.debian.tar.gz | tail -1)
dch -i
dpkg-buildpackage

Any suggestions for improvement will be welcome, I don’t claim to be the world’s greatest shell scripter. But please note that I generally aim to write shell scripts that can be understood by people who aren’t experts. So if you can replace the program with a single line of Perl I will be impressed but I won’t implement your solution.

ZFS on Debian/Wheezy

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable.

BTRFS vs ZFS

Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don’t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don’t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity.

I am about to install a system to be a file server and Xen server which probably isn’t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won’t be upgraded a lot I’ll start with Debian/Wheezy even though it isn’t released yet because the system will be in use without much change well after Squeeze security updates end.

ZFS on Wheezy

Getting ZFS to basically work isn’t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn’t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn’t particularly difficult but it’s a pity that the zfsonlinux site didn’t support recent kernels.

Root on ZFS

The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device.

Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn’t ideal, but when you have an OS image that is backed up (and not so important) and a data store that’s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it.

There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort.

The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn’t work with the Debian initramfs for some reason and modules failed to load. This wasn’t necessarily a show-stopper as I can modify such things myself, but it’s another painful thing to manage and another way that the system can potentially break on upgrade.

The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the “zfs set mountpoint” command. This of course means that it doesn’t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn’t a good option. So for this new server I’ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else.

Correct Alignment

After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors.

Installing ZFS

deb http://www.coker.com.au wheezy zfs

I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn’s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik’s binary packages but I think it’s best to rebuild Ubuntu packages to use on Debian.

The server in question hasn’t gone live in production yet (it turns out that we don’t have agreement on what the server will do). But so far it seems to be working OK.

New SE Linux Policy for Wheezy

I’ve just uploaded a new SE Linux policy for Debian/Wheezy. It now works correctly with systemd and Chromium, two significant features that I wanted for Wheezy. Now it turns out that we have until the end of the month for Wheezy updates, so I may get another version of the policy uploaded before then. If so it will only be for relatively minor changes, I think that most SE Linux users would be reasonably happy with policy the way it is. Anything that doesn’t work now can probably be solved by local configuration changes.

execmem

The current version of KDE in Debian is 4.8.4, it seems that large parts of the KDE environment depend on execmem access, this includes kwin and plasma-desktop. Basically there is no possibility of having a KDE desktop environment without those programs and therefore KDE depends on execmem access.

Debugging this is difficult as the important programs SEGV when denied execmem access and the KDE crash handler really gets in the way of debugging it – running /usr/bin/plasma-desktop results in the process forking a child and detaching from the gdb session.

The most clear example of an execmem issue in KDE is from the program /usr/lib/kde4/libexec/kwin_opengl_test which gives the following error:
LLVM ERROR: Allocation failed when allocating new memory in the JIT
Can’t allocate RWX Memory: Permission denied

To make this work you run the command “setsebool -P allow_execmem 1” which gives many domains the ability to create writable-executable memory regions.

I raised this issue for discussion on the SE Linux mailing list and Hinnerk van Bruinehsen wrote an informative message in response summarising the situation [1]. It seems that it’s possible to compile some of the programs in question to not use the JIT and therefore not require such access and there is a build option in Gentoo to allow it. But it’s impractically difficult for me to fork KDE in Debian so the only option is to recommend that people enable the allow_execmem boolean for Debian desktop systems running SE Linux.

Debian SE Linux Status June 2012

It’s almost the Wheezy freeze time and I’ve been working frantically to get things working properly.

Policy Status

At the moment I’m preparing an upload of the policy which will support KDE (and probably most desktop environment) logins and many little fixes related to server operations (particularly MTAs). I would like to get another version done before Wheezy is released, but if Wheezy releases with version 2.20110726-6 of the policy that will be OK. It will work well enough for most things that users will be able to use local changes for the things that don’t work.

One significant lack with the current policy is that systemd won’t work. I’ve included most of the policy changes needed, but haven’t done any of the testing and tweaking that is necessary to make it work properly.

I would like to see policy support for systemd in a Wheezy update if I don’t get it done in time for the first release. If I don’t get it done in time for the release and if the release team don’t accept it for an update then I’ll put it in my own repository so anyone who needs it can get it.

/run Labelling

One significant change for Wheezy is to use a tmpfs mounted on /run instead of /var/run. This means that lots of daemon start scripts create subdirectories of /run at boot time which need to have SE Linux labels applied for correct operation. The way things work is that usually the daemon will write to the directory immediately after the init script has created it, so I can’t just have my own script recursively relabel all of /run.

Some packages that need to be patched are x11-common #677831, clamav-daemon #677686, sasl2-bin #677685, dkim-filter #677684, and cups #677580. I am sure that there are others.

[ -x /sbin/restorecon ] && /sbin/restorecon -R $DIR

Generally if you are writing an init script and creating a directory under /run then you need to have some shell code like the above immediately after it’s created. Also the same applies for directories under /tmp and any other significant directories that are created at boot time.

Upgrading

Currently there are some potential problems with the upgrade process, I’m working on them at the moment. Ideally an “apt-get dist-upgrade” would cleanly upgrade everything. But at the moment it seems likely that the upgrade might initially go wrong and then work on the second try. There are some complications such as the selinux-policy-default package owning a config file which is used by mcstransd (which is part of the policycoreutils package), when the config file format changes you get order dependencies for the upgrade.

Kernel Support

My aim when developing a new SE Linux release for Debian is that the policy should work as much as possible with the user-space from the previous release. So if you upgrade from Squeeze to Wheezy you should be able to start the process by upgrading the SE Linux policy (which drags in the utilities and lots of libraries). This means that if you have a server running you don’t have to put it out of action for the entire upgrade, you can get the policy going and then get other things going. I haven’t tested this yet but I don’t expect any problems (apart from all the dependencies).

Also the policy should work with the kernel from the previous release. So if you have a virtual server where it’s not convenient to upgrade the kernel then that shouldn’t stop you from upgrading the user-space and the SE Linux policy. I’ve tested this and found one bug, the sepolgen-ifgen utility that you need to run before audit2allow -R won’t work if the kernel is older than the utilities #677730. I don’t know if it will be possible to get this fixed. Anyway it’s not that important, you can always copy the audit log to another system running the same policy to run audit2allow, it’s not convenient but not THAT difficult either.

The End Result

I think that the result of using SE Linux in Wheezy will be quite good for the people who get the upgrade done and who modify a few init scripts that don’t get the necessary changes in time. I anticipate that someone who doesn’t know much about SE Linux will be able to get a basic workstation or small server installation done in considerably less than an hour if they read the documentation and someone who knows what they are doing will get it done in a matter of minutes (plus download and install time which can be significant on old hardware).

At the moment I’m in the process of upgrading all of my systems to Unstable (currently Testing has versions of some SE Linux packages that are too broken). While doing this I will keep discovering bugs and fix as many of them as possible. But it seems that I’ve already fixed most things that affect common users.

Also BTRFS works well. Not that supporting a new filesystem is a big deal (all that’s needed is XATTR support), but having all the nice new features on one system is a good thing. Now I just need to get systemd working.

SE Linux Status in Debian 2012-03

I have just finished updating the user-space SE Linux code in Debian/Unstable to the version released on 2012-02-16. There were some changes to the build system from upstream which combined with the new Debian multi-arch support involved a fair bit of work for me. While I was at it I converted more of them to the new Quilt format to make it easier to send patches upstream. In the past I have been a bit slack about sending patches upstream, my aim for the next upstream release of user-space is to have at least half of my patches included upstream – this will make things easier for everyone.

Recently Mika Pflüger and Laurent Bigonville have started work on Debian SE Linux, they have done some good work converting the refpolicy source (which is used to build selinux-policy-default) to Quilt. Now it will be a lot easier to send policy patches upstream and porting them to newer versions of the upstream refpolicy.

Now the next significant thing that I want to do is to get systemd working correctly with SE Linux. But first I have to get it working correctly wit cryptsetup.

SE Linux Status in Debian 2012-01

Since my last SE Linux in Debian status report [1] there have been some significant changes.

Policy

Last year I reported that the policy wasn’t very usable, on the 18th of January I uploaded version 2:2.20110726-2 of the policy packages that fixes many bugs. The policy should now be usable by most people for desktop operations and as a server. Part of the delay was that I wanted to include support for systemd, but as my work on systemd proceeded slowly and others didn’t contribute policy I could use I gave up and just released it. Systemd is still a priority for me and I plan to use it on all my systems when Wheezy is released.

Kernel

Some time between Debian kernel 3.0.0-2 and 3.1.0-1 support for an upstream change to the security module configuration was incorporated. Instead of using selinux=1 on the kernel command line to enable SE Linux support the kernel option is security=selinux. This change allows people to boot with security=tomoyo or security=apparmor if they wish. No support for Smack though.

As the kernel silently ignores command line parameters that it doesn’t understand so there is no harm in having both selinux=1 and security=selinux on both older and newer kernels. So version 0.5.0 of selinux-basics now adds both kernel command-line options to GRUB configuration when selinux-activate is run. Also when the package is upgraded it will search for selinux=1 in the GRUB configuration and if it’s there it will add security=selinux. This will give users the functionality that they expect, systems which have SE Linux activated will keep running SE Linux after a kernel upgrade or downgrade! Prior to updating selinux-basics systems running Debian/Unstable won’t work with SE Linux.

As an aside the postinst file for selinux-basics was last changed in 2006 (thanks Erich Schubert). This package is part of the new design of SE Linux in Debian and some bits of it haven’t needed to be changed for 6 years! SE Linux isn’t a new thing, it’s been in production for a long time.

Audit

While the audit daemon isn’t strictly a part of SE Linux (each can be used without the other) it seems that most of the time they are used together (in Debian at least). I have prepared a NMU of the new upstream version of audit and uploaded it to delayed/7. I want to get everything related to SE Linux up to date or at least with comparable versions to Fedora. Also I sent some of the Debian patches for the auditd upstream which should reduce the maintenance effort in future.

Libraries

There have been some NMUs of libraries that are part of SE Linux. Due to a combination of having confidence in the people doing the NMUs and not having much spare time I have let them go through without review. I’m sure that I will notice soon enough if they don’t work, my test systems exercise enough SE Linux functionality that it would be difficult to break things without me noticing.

Play Machine

I am now preparing a new SE Linux “Play Machine” running Debian/Unstable. I wore my Play Machine shirt at LCA so I’ve got to get one going again soon. This is a good exercise of the strict features of SE Linux policy, I’ve found some bugs which need to be fixed. Running Play Machines really helps improve the overall quality of SE Linux.