Archives

Categories

DNS Secondaries and Web Security

At the moment there are ongoing security issues related to web based services and DNS hijacking. the Daily Ack has a good summary of the session hijacking issue [1].

For a long time it has been generally accepted that you should configure a DNS server to not allow random machines on the Internet to copy the entire zone. Not that you should have any secret data there anyway, but it’s regarded as just a precautionary layer of security by obscurity.

Dan Kaminsky (who brought the current DNS security issue to everyone’s attention) has described some potential ways to alleviate the problem [2]. One idea is to use random case in DNS requests (which are case insensitive but case preserving), so if you were to lookup wWw.cOkEr.CoM.aU and the result was returned with different case then you would know that it was forged.

Two options which have been widely rejected are using TCP for DNS (which is fully supported for the case where an answer can not fit in a single UDP packet) and sending requests twice (to square the number of combinations that would need to be guessed). They have been rejected due to the excessive load on the servers (which are apparently already near capacity).

One option that does not seem to get mentioned is the possibility to use multiple source IP addresses, so instead of merely having 2^16 ports to choose from you could multiply that by as many IP addresses as you have available. In the past I’ve worked for ISPs that could have dedicated a /22 (1024 IP addresses) to their DNS proxy if it would have increased the security of their customers – an ISP of the scale that has 1024 spare IP addresses available is going to be a major target of such attacks! Also with some fancy firewall/router devices it would not be impossible to direct all port 53 traffic through the DNS proxies. That would mean that an ISP with 200,000 broadband customers online could use a random IP address from that pool of 200,000 IP addresses for every DNS request. While attacking a random port choice out of 65500 ports is possible, if it was 65500 ports over a pool of 200,000 IP addresses it would be extremely difficult (I won’t claim it to be impossible).

One problem with the consideration that has been given to TCP is that it doesn’t account for the other uses of TCP, such as for running DNS secondaries.

In Australia we have two major ISPs (Telstra and Optus) and four major banks (ANZ, Commonwealth, NAB, and Westpac). It shouldn’t be difficult for arrangements to be made for the major ISPs to have their recursive DNS servers (the caching servers that their customers talk to) act as slaves for the DNS zones related to those four banks (which might be 12 zones or more given the use of different zones for stock-broking etc). If that was combined with a firewall preventing the regular ISP customers (the ones who are denied access to port 25 to reduce the amount of spam) from receiving any data from the Internet with a source port of 53 then the potential for attacks on Australian banks would be dramatically decreased. I note that the Westpac bank has DNS secondaries run by both Optus and Telstra (which makes sense for availability reasons if nothing else), so it seems that the Telstra and Optus ISP services could protect their customers who use Westpac without any great involvement from the bank.

Banks have lots of phone lines and CTI systems. It would be easy for each bank to have a dedicated phone number (which is advertised in the printed phone books, in the telephone “directory assistance” service, and in brochures available in bank branches – all sources which are more difficult to fake than Internet services) which gave a recorded message of a list of DNS zone names and the IP addresses for the master data. Then every sysadmin of every ISP could mirror the zones that would be of most use to their customers.

Another thing that banks could do would be to create a mailing list for changes to their DNS servers for the benefit of the sysadmins who want to protect their customers. Signing mail to such a list with a GPG key and having the fingerprint available from branches should not be difficult to arrange.

Another possibility would be to use the ATM network to provide security relevant data. Modern ATMs have reasonably powerful computers which are used to display bank adverts when no-one is using them. Having an option to press a button on the ATM to get a screen full of Internet banking security details of use to a sysadmin should be easy to implement.

For full coverage (including all the small building societies and credit unions) it would be impractical for every sysadmin to have a special case for every bank. But again there is a relatively easy solution. A federal agency that deals with fraud could maintain a list of zone names and master IP addresses for every financial institution in the country and make it available on CD. If the CD was available for collection from a police station, court-house, the registry of births, deaths, and marriages, or some other official government office then it should not have any additional security risks. Of course you wouldn’t want to post such CDs, even with public key signing (which many people don’t check properly) there would be too much risk of things going wrong.

In a country such as the US (which has an unreasonably large number of banks) it would not be practical to make direct deals between ISPs and banks. But it should be practical to implement a system based on a federal agency distributing CDs with configuration files for BIND and any other DNS servers that are widely used (is any other DNS server widely used?).

Of course none of this would do anything about the issue of Phishing email and typo domain name registration. But it would be good to solve as much as we can.

Ownership of the Local SE Linux Policy

A large part of the disagreement about the way to manage the policy seems to be based on who will be the primary “owner” of the policy on the machine. This isn’t a problem that only applies to SE Linux, the same issue applies for various types of configuration files and scripts throughout the process of distribution development. Having a range of modules which can be considered configuration data that come from a single source seems to make SE Linux policy unique among other packages. The reasons for packaging all Apache modules in the main package seem a lot clearer.

One idea that keeps cropping up is that as the policy is modular it should be included in daemon packages and the person maintaining the distribution package of the policy should maintain it. The reason for this request seems to usually be based on the idea that the person who packages a daemon for a distribution knows more about how it works than anyone else, I believe that this is false in most cases. When I started working on SE Linux I had a reasonable amount of experience in maintaining Debian packages of daemons and server processes, but I had to learn a lot about how things REALLY work to be able to write good policy. Also if we were to have policy modules included in the daemon packages, then those packages would need to be updated whenever there were serious changes to the SE Linux policy. For example Debian/Unstable flip-flopped on MCS support recently, changing the policy packages to re-enable MCS was enough pain, getting 50 daemon packages updated would have been unreasonably painful. Then of course there is the case where two daemons need to communicate, if the interface which is provided with one policy module has to be updated before another module can be updated and they are in separate packages then synchronised updates to two separate packages might be required for a single change to the upstream policy. I believe that the idea of having policy modules owned by the maintainers of the various daemon packages is not viable. I also believe that most people who package daemons would violently oppose the idea of having to package SE Linux policy if they realised what would be required of them.

Caleb Case seems to believe that ownership of policy can either be based on the distribution developer or the local sys-admin with apparently little middle-ground [1]. In the section titled “The Evils of Single Policy Packages” he suggests that if an application is upgraded for a security fix, and that upgrade requires a new policy, then it requires a new policy for the entire system if all the policy is in the same package. However the way things currently work is that upgrading a Debian SE Linux policy package does not install any of the new modules. They are stored under /usr/share/selinux/default but the active modules are under /etc/selinux/default/modules/active. An example of just such an upgrade is the Debian Security Advisory DSA-1617-1 for the SE Linux policy for Etch to address the recent BIND issue [2]. In summary the new version of BIND didn’t work well with the SE Linux policy, so an update was released to fix it. When the updated SE Linux policy package is installed it will upgrade the bind.pp module if the previous version of the package was known to have the version of bind.pp that didn’t allow named to bind() to most UDP ports – the other policy modules are not touched. I think that this is great evidence to show that the way things currently work in Debian work well. For the hypothetical case where a user had made local modifications to the bind.pp policy module, they could simply put the policy package on hold – I think it’s safe to assume that anyone who cares about security will read the changelogs for all updates to released versions of Debian, so they would realise the need to do this.

Part of Caleb’s argument rests on the supposed need for end users to modify policy packages (IE to build their own packages from modified source). I run many SE Linux machines, and since the release of the “modular” policy (which first appeared in Fedora Core 5, Debian/Etch, and Red Hat Enterprise Linux 5) I have never needed to make such a modification. I modify policy regularly for the benefit of Debian users and have a number of test machines to try it out. But for the machines where I am a sysadmin I just create a local module that permits the access that is needed. The only reason why someone would need to modify an existing module is to remove privileges or to change automatic domain transition rules. Changing automatic domain transitions is a serious change to the policy which is not something that a typical user would want to do – if they were to do such things then they would probably grab the policy source and rebuild all the policy packages. Removing privileges is not something that a typical sysadmin desires, the reference policy is reasonably strict and users generally don’t look for ways to tighten up the policy. In almost all cases it seems best to consider that the policy modules which are shipped by the distribution are owned by the distribution not the sysadmin. The sysadmin will decide which policy modules to load, what roles and levels to assign to users with the semanage tool, and what local additions to add to the policy. For the CentOS systems I run I use the Red Hat policy, I don’t believe that there is a benefit for me to change the policy that Red Hat ships, and I think that for people who have less knowledge about SE Linux policy than me there are more reasons not to change such policy and less reasons to do so.

Finally Caleb provides a suggestion for managing policy modules by having sym-links to the modules that you desire. Of course there is nothing preventing the existence of a postfix.pp file on the system provided by a package while there is a local postfix.pp file which is the target of the sym-link (so the sym-link idea does not support the idea of having multiple policy packages). With the way that policy modules can be loaded from any location, the only need for sym-links is if you want to have an automatic upgrade script that can be overridden for some modules. I have no objection to adding such a feature to the Debian policy packages if someone sends me a patch.

Caleb also failed to discuss how policy would be initially loaded if packaged on a per-module basis. If for example I had a package selinux-policy-default-postfix which contains the file postfix.pp, how would this package get installed? I am not aware of the Debian package dependencies (or those of any other distribution) being about to represent that the postfix package depends on selinux-policy-default-postfix if and only if the selinux-policy-default package is installed. Please note that I am not suggesting that we add support for such things, a package management system that can solve Sudoku based on package dependency rules is not something that I think would be useful or worth having. As I noted in my previous post about how to package SE Linux policy for distributions [3] the current Debian policy packages have code in the postinst (which I believe originated with Erich Schubert) to load policy modules that match the Debian packages on the system. This means that initially setting up the policy merely requires installing the selinux-policy-default package and rebooting. I am inclined to reject any proposed change which makes the initial install of of the policy more difficult than this.

After Debian/Lenny is released I plan to make some changes to the policy. One thing that I want to do is to have a Debconf option to allow users to choose to automatically upgrade their running policy whenever they upgrade the Debian policy package, this would probably only apply to changes within one release (IE it wouldn’t cause an automatic upgrade from Lenny+1 policy to Lenny+2). Another thing I would like to do is to have the policy modules which are currently copied to /etc/selinux/default/modules/active instead be hard linked when the source is a system directory. That would save about 12M of disk space on some of my systems.

I’ve taken the unusual step of writing two blog posts in response to Caleb’s post not because I want to criticise him (he has done a lot of good work), but because he is important in the SE Linux community and his post deserves the two hours I have spent writing responses to it. While writing these posts I have noticed a number of issues that can be improved, I invite suggestions from Caleb and others on how to make such improvements.

SE Linux Policy Packaging for a Distribution

Caleb Case (Ubuntu contributer and Tresys employee) has written about the benefits of using separate packages for SE Linux policy modules [1].

Firstly I think it’s useful to consider some other large packages that could be split into multiple packages. The first example that springs to mind is coreutils which used to be textutils, shellutils, and fileutils. Each of those packages contained many programs and could conceivably have been split. Some of the utilities in that package are replaced for most use, for example no-one uses the cksum utility, generally md5sum and sha1sum (which are in the same package) are used instead. Also the pinky command probably isn’t even known by most users who use finger instead (apart from newer Unix users who don’t even know what finger is). So in spite of the potential benefit of splitting the package (or maintaining the previous split) it was decided that it would be easier for everyone to have a single package. The merge of the three packages was performed upstream, but there was nothing preventing the Debian package maintainer from splitting the package – apart from the inconvenience to everyone. The coreutils package in Etch takes 10M of disk space when installed, as it’s almost impossible to buy a new hard drive smaller than 80G that doesn’t seem to be a problem for most users.

The second example is the X server which has separate packages for each video card. One thing to keep in mind about the X server is that the video drivers don’t change often. While it is quite possible to remove a hard drive from one machine and install it in another, or duplicate a hard drive to save the effort of a re-install (I have done both many times) they are not common operations in the life of a system. Of course when you do require such an update you need to first install the correct package (out of about 60 choices), which can be a challenge. I suspect that most Debian systems have all the video driver packages installed (along with drivers for wacom tablets and other hardware devices that might be used) as that appears to be the default. So it seems likely that a significant portion of the users have all the packages installed and therefore get no benefit from the split package.

Now let’s consider the disk space use of the selinux-policy-default package – it’s 24M when installed. Of that 4.9M is in the base.pp file (the core part of the policy which is required), then there’s 848K for the X server (which is going to be loaded on all Debian systems that have X clients installed – due to an issue with /tmp/.ICE-unix labelling [2]). Then there’s 784K for the Postfix policy (which is larger than it needs to be – I’ve been planning to fix this for the past four years or so) and 696K for the SSH policy (used by almost everyone). The next largest is 592K for the Unconfined policy, the number of people who choose not to use this will be small, and as it’s enabled by default it seems impractical to provide a way of removing it.

One possibility for splitting the policy is to create a separate package of modules used for the less common daemons and services, if modules for INN, Cyrus, distcc, ipsec, kerberos, ktalk, nis, PCMCIA, pcscd, RADIUS, rshd, SASL, and UUCP were in a separate package then that would reduce the installed size of the main package by 1.9M while providing no change in functionality to the majority of users.

One thing to keep in mind is that each package at a minimum will have a changelog and a copyright file (residing in a separate directory under /usr/share/doc) and three files as part of the dpkg data store, each of which takes up at least one allocation unit on disk (usually 4K). So adding one extra package will add at least 24K of disk space to every system that installs it (or 32K if the package has postinst and postrm scripts). This is actually a highly optimal case, the current policy packages (selinux-policy-default and selinux-policy-mls) each take 72K of disk space for their doc directory.

One of my SE Linux server sytems (randomly selected) has 23 policy modules installed, if they were in separate packages there would be a minimum of 552K of disk space used by packaging, 736K if there were postinst and postrm scripts, and as much as 2M if the doc directory for each package was similar to the current doc directories). As the system in question needs 5796K of policy modules, the 2M of overhead would make it approach 8M of disk space. So it would only be a saving of 16M over the current situation. While saving that amount of disk space is a good thing, I think that when balanced against the usability issues it’s not worth-while.

Currently the SE Linux policy packages will determine what applications are installed and automatically load policy packages to match. I don’t believe that it’s possible to have a package post-inst script install other packages (and if it is possible I don’t think it’s desirable). Therefore to have separate packages would make a significant difference to the ease of use, it seems that the best way to manage it would be to have the core policy package include a script to install the other packages.

Finally there’s the issue of when you recognise the need for a policy module. It’s not uncommon for me to do some work for a client while on a train, bus, or plane journey. I will grab packages needed to simulate a configuration that the client desires and then work out how to get it going correctly while on the journey. While it would not be a problem for me (I always have the SE Linux policy source and all packages on hand) I expect that many people who have similar needs might find themself a long way from net access without the policy package that they need to do their work. Sure such people could do their work in permissive mode, but that would encourage them to deploy in permissive mode too and thus defeat the goals of the SE Linux project (in terms of having wide-spread adoption).

My next post on this topic will cover the issue of custom policy.

Updated to note that Caleb is a contributor to Ubuntu not a developer.

Australian Business and IT Expo

I’ve just visited the Australian Business and IT Expo (ABITE) [1]. I haven’t been to such an event for a while, but Peter Baker sent a link for a free ticket to the LUV mailing list and I was a bit bored to I attended.

The event was a poor shadow on previous events that I had attended. The exhibition space was shared with an event promoting recreational activities for retirees, and an event promoting wine and gourmet food. I’m not sure why the three events were in the room, maybe they figured that IT people and senior citizens both like gourmet food and wine.

The amount of space used for the computer stands was small, and there was no great crowd of delegates – when they can’t get a good crowd for Saturday afternoon it’s a bad sign for the show.

I have previously blogged about the idea of putting advertising on people’s butts [2]. One company had two women working on it’s stand with the company’s name on the back of their shorts.

A representative of a company in the business of Internet advertising asked me how many hits I get on my blog, I told him 2,000 unique visitors a month (according to Webalizer) which seemed to impress him. Actually it’s about 2,000 unique visitors a day. I should rsync my Webalizer stats to my EeePC so I can give detailed answers to such questions.

The IT event seemed mostly aimed at managers. There were some interesting products on display, one of which was a device from TabletPC.com.au which had quite good handwriting recognition (but the vocabulary seemed limited as it couldn’t recognise a swear-word I used as a test).

Generally the event was fun (including the wine and cheese tasting) and I don’t regret going. If I had paid $10 for a ticket I probably would have been less happy with it.

Updated to fix the spelling of “wine”. Not a “wind tasting”.

Starting to Blog

The best way to run a blog is to run your own blog server. This can mean running an instance on someone else’s web server (some ISPs have special hosting deals for bloggers on popular platforms such as WordPress), but usually means having shell access to your own server (I’ve previously written about my search for good cheap Xen hosting [1]).

There are platforms that allow you to host your own blog without any technical effort. Three popular ones are WordPress.com, LiveJournal.com, and Blogger.com. But they give you less control over your own data, particularly if you don’t use your own DNS name (blogger allows you to use their service with your own DNS name).

Currently it seems to me that WordPress is the best blog software by many metrics. It has a good feature set, a plugin interface with lots of modules available, and the code is free. The down-side is that it’s written in PHP and has the security issues that tend to be associated with large PHP applications.

Here is a good summary of the features of various blog server software [2]. One that interests me is Blojsom – a blog server written in Java [3]. The Java language was designed in a way that leads to less risk of security problems than most programming languages, as it seems unlikely that anyone will write a Blog server in Ada it seems that Java is the best option for such things. I am not planning to switch, but if I was starting from scratch I would seriously consider Blojsom.

But for your first effort at blogging it might be best to start with one of the free hosted options. You can always change later on and import the old posts into your new blog. If you end up not blogging seriously then using one of the free hosted services saves you the effort of ongoing maintenance.

CPU vs RAM

When configuring servers the trade-offs between RAM and disk are well known. If your storage is a little slow then you can often alleviate the performance problems by installing more RAM for caching and to avoid swapping. If you have more than adequate disk IO capacity then you can over-commit memory and swap out the things that don’t get used much.

One that often doesn’t get considered i the trade-off between RAM and CPU. I just migrated a server image from a machine with two P4 CPUs to a DomU on a machine with Opteron CPUs. The P4 system seemed lightly loaded (a maximum of 30% CPU time in use over any 5 minute period) so I figured that if two P4 CPUs are 30% busy then a single Opteron core should do the job. It seems that when running 32bit code, 30% of 2*3.0GHz P4 CPUs is close to the CPU power of one core of an Opteron 2352 (2.1GHz). I’m not sure whether this is due to hyper-threading actually doing some good or inefficiencies in running 32bit code on the Opteron – but the Opteron is not giving the performance I expected in this regard.

Now having about 90% of the power of that CPU core in use might not be a problem, except that the load came in bursts. When a burst took the machine to 100% CPU power a server process kept forking off children to answer requests. As all the CPU power was being used it took a long time to answer queries (several seconds) so the queue started growing without end. Eventually there were enough processes running that all memory was used, the machine started thrashing, and eventually the kernel out of memory handler started killing things.

I rebooted the DomU with two VCPUs (two Opteron cores) and there was no problem, performance was good, and because the load bursts last less than a minute the load average seems to stay below 1.

It seems that the use of virtual machines increases the scope of this problem. The advantage of virtual machines is that you can add extra virtual hardware more easily (up to the limit of the physical hardware of course) – I could give the DomU in question 6 Opteron cores in a matter of minutes if it was necessary. The disadvantage is that the CPU use of other virtual machines can impact the operation. As there seems to be an exponential relationship between the number of CPU cores in a system and the overall price it’s not feasible to just put in 32 core Xen servers. While CPU power has generally been increasing faster than disk performance for a long time (at least the last 20 years) it seems that virtualisation provides a way of using a lot of that CPU power. It is possible to have a 1:1 mapping of real CPUs and VCPUs in the Xen DomU’s, if you were to install 8 DomU’s that each had one VCPU on a server with 8 cores then there would be no competition between DomUs for CPU time – but that would significantly increase the cost of running them (some ISPs offer this for a premium price).

In this example, if I had a burst of load for the service in question at the same time as other DomUs were using a lot of CPU time (which is a possibility as the other DomUs are the clients for the service in question) then I might end up with the same problem in spite of having assigned two VCPUs to the DomU.

The real solution is to configure the server to limit the number of children that it forks off, the limit can be high enough to guarantee 100% CPU use at times of peak load without being high enough to start swapping.

I wonder how this goes with ISPs that offer Xen hosting. It seems that you would only need to have one customer who shares the same Xen server as you experiencing such a situation to cause enough disk IO to cripple the performance that you get.

SpamAssassin During SMTP

For some time people have been telling me about the benefits of SpamAssassin (SA). I have installed it once for a client (at their demand and against my recommendation) but was not satisfied with the result (managing the spam folder was too complex for their users).

The typical configuration of SA has it run after mail has been accepted by the server. Messages that it regards as spam are put into a spam folder. This means that when someone phones you about some important message you didn’t receive then you have to check that folder. Someone who sends mail to a user who has such a SA configuration can not expect that the message will either be received or rejected (thus giving them a bounce message).

Even worse it seems to be quite common for technical users to train the Bayesian part of SA on messages from the spam folder – without reviewing them! Submitting a folder of spam that has been carefully reviewed for Bayesian training can increase the accuracy of classification (including taking account for locality and language differences in spam). Submitting a folder which is not reviewed means that when a false-positive gets into that folder (which will eventually happen) it is used as training for spam recognition thus increasing the incidence of false-positives!

Spam has been becoming more of a problem for me recently, on a typical day between 20 and 40 spam messages would get past the array of DNSBL services I use and be re-sent to pass the grey-listing. Also I have been receiving complaints from people who want to send email to me about some of the DNSBL and RHSBL services I use (the rfc-ignorant.org service gets a lot of complaints – there are a huge number of ignorant and lazy people running mail servers).

So now I have installed spamassassin-milter to have SA run during the SMTP protocol. Then if SA checks indicate that the message is SPAM my mail server can just reject the message with a 55x which will cause the sending mail server to generate a local bounce (if it’s a legitimate message) or to just be discard it in the case of a spam server. Here is how to set it up on Debian/Lenny and CentOS 5:

Install the package yum install spamass-milter or apt-get install spamass-milter spamassassin spamc (spamassassin seems to be installed by default on CentOS). On a Debian system the milter will be setup and running. On CentOS you have to run the following commands:
useradd -m -c "Spamassassin Milter" -s /bin/false spamass-milter
mkdir /var/run/spamass-milter
chown spamass-milter /var/run/spamass-milter
chmod 711 /var/run/spamass-milter
echo SOCKET="/var/run/spamass-milter/spamass.sock" >> /etc/sysconfig/spamass-milter

On CentOS edit /etc/init.d/spamass-milter and change the daemon start line to ‘runuser – spamass-milter -s /bin/bash -c "/usr/sbin/spamass-milter -p $SOCKET -f $EXTRA_FLAGS"‘ Then add the following lines below it:
chown postfix:postfix /var/run/spamass-milter/spamass.sock
chmod 660 /var/run/spamass-milter/spamass.sock

The spamass-milter program talks to the SpamAssassin daemon spamd.

On both Debian and CentOS run the command “useradd -c Spamassassin -m -s /bin/false spamassassin” to create an account for SA. The Debian bug #486914 [1] has a request to have SA not run as root by default.

On CentOS it seems that SA wants to use a directory under the spamass-milter home directory, the following commands alllow this. It would be good to have it not do that, or maybe it would be better to have the one Unix account used for SA and the milter.
chmod 711 ~spamass-milter
mkdir ~spamassassin/.spamassassin
chown spamassassin ~spamassassin/.spamassassin

On Debian edit the file /etc/default/spamassassin and add “-u spamassassin -g spamassassin” to the OPTIONS line. On CentOS edit the file /etc/sysconfig/spamassassin and add “-u spamassassin -g spamassassin” to the SPAMDOPTIONS line.

To enable the daemons, on CentOS you need to run “chkconfig spamass-milter on ; chkconfig spamassassin on“, on Debian edit the file /etc/default/spamassassin and set ENABLED=1.

Now start the daemons, on CentOS use the command “service spamassassin start ; service spamass-milter start“, on Debian use the command “/etc/init.d/spamassassin start“.

Now you have to edit the mail server configuration, for Postfix on CentOS the command “postconf -e smtpd_milters=unix:/var/run/spamass-milter/spamass.sock” will do it, for Postfix on Debian the command “postconf -e smtpd_milters=unix:/var/spool/postfix/spamass/spamass.sock” will do it.

Now restart Postfix and it should be working.

For correct operation you need to ensure that the score needed for a bounce is specified as the same number in both the spamass-milter and SA configuration. If you have a lower number for the spamass-milter configuration (as is the default in Debian) then bounces can be generated – you should never generate a bounce for a spam. The config file /etc/default/spamass-milter allows you to specify the score for rejecting mail, I am currently using a score of 5. Any changes to the score need matching changes to /etc/mail/spamassassin/local.cf (which has a default required_score of 5 in Debian).

You can grep for “spamd..result..Y” in your mail log to see entries for messages that were rejected.

One problem that I have with this configuration on Debian (not on CentOS) is that spamd is logging messages such as “spamd: handle_user unable to find user: ‘russell’“. I don’t want it to look for ~russell when processing mail for russell@coker.com.au because I have a virtual domain set up and the delivery mailbox has a different name. Ideally I could configure it to know the mapping between users and mailboxes (maybe by parsing the /etc/postfix/virtual.db file). But having it simply not attempt to access per-user configuration would be good too. Any suggestions would be appreciated.

Now that I have SpamAssassin running it seems that I am getting about 5 spams a day, the difference is significant. The next thing I will do is make some of the DNSBL checks that are prone to false-positives become SpamAssassin scores instead.

When I started writing this post I was not planning to compare the sys-admin experiences of CentOS and Debian. But it does seem that there is less work involved in the task of installing Debian packages.

Compassion for Windows Users

In a discussion which covered some of the differences between Linux and Windows, a Windows using friend asked me if I felt compassion for Windows users.

I feel some compassion for people who have bad working environments. While using an operating system that has poor support for the business tasks does decrease the quality of the working environment, there are bigger issues. For example a while ago I was doing some sys-admin work for a financial organisation. I had to use Windows for running the SSH client to connect to Linux servers, this was annoying and decreased my productivity due to the inability to script connections etc. My productivity was also decreased because of my unfamiliarity with the Windows environment, it seems reasonable to assume that when you hire a Linux sys-admin they will have some experience of Linux on the desktop and be quite productive with a Linux desktop system – while the same can not be said for a Windows desktop. But what really made the working environment awful was the paperwork and the procedures. If a server doesn’t work properly and someone says “please fix it now” and I only have a VT100 terminal then I’ll be reasonably happy with that work environment (really – I wouldn’t mind a contract where the only thing on my desk was a VT100 connected to a Linux server). But when a server process hangs in the same way several times a week, when the cause of the problem is known and the fix (restarting the process) is known it really pains me to have to wait for a management discussion about the scope of the problem before restarting it.

But I don’t have a great deal of sympathy for people who end up in bad working environments such as the one I was briefly in. Anyone who is capable of getting such a job is capable of getting a job with a better working environment while still earning significantly more than the median income. The people I feel sorry for are the ones who work on the minimum wage. I don’t think that the difference between Linux and Windows on the desktop would matter much if you were getting the minimum wage, and people who are on the minimum wage don’t have a lot of choice in regard to employment (I think that all options for them suck).

I don’t have much sympathy for adults who use Windows at home. I have to admit that there are some benefits to running Windows at home, mainly that the hardware vendors support it better (few companies sell PCs with Linux pre-loaded) and there are some commercial games which are in some ways better than the free games (of course there are more than enough free Linux games to waste all your time – and some games are best suited to a console). Linux has significantly lower hardware requirements than Windows (my main machine which I am using to write this blog post is more than three years old and has less power than any other machine on sale today apart from some ultra-mobile PCs), so any long-term Windows user can install Linux on one of their machines which lacks the resources to run the latest version of Windows.

The only Windows users for whom I have much sympathy are children. When I was young every PC came with a BASIC interpreter and everyone shared source code. Books were published which taught children how to program in BASIC which included fairly complete example programs. For the cases where proprietary software was needed the prices used to be quite low (admittedly the programs were much less complex – so pricing is probably in line with the effort or writing the code). Now it seems that computers are often being provided to children as closed systems that they can’t manipulate, the web browser has replaced the TV.

I believe that Linux is the ideal OS for a child to use. There is a wide range of free educational programs (including kturtle – the traditional Logo turtle) and there are also a range of free powerful programs which can be used by any child. Few parents would buy Photoshop or Adobe Illustrator for a child to play with, but anyone can give a child a $100 PC with GIMP and Inkscape installed. They might as well give 3yo children access to the GIMP – it will be less messy than fingerpainting!

I expect that some parents would not consider Linux for their children because they don’t know how to use it. Fortunately Linux is easy enough to use that a child can install it without effort. Some time ago the 11yo daughter of a friend who was visiting asked if she could play some computer games. I gave her a Fedora CD and one of the PCs from my test lab and told her that she had to install the OS first. Within a small amount of time she had Fedora installed and was playing games. While the games she played were not particularly educational, the fact that she could install the OS on a computer was a useful lesson.

It seems to me that children who are raised as Windows users are less likely to learn how computers work or be able to control them properly. I expect that on average a child who is raised in such a manner will have fewer career options in today’s environment than one who properly understands and controls computers.

Executable Stacks in Lenny

One thing that I would like to get fixed for Lenny is the shared objects which can reduce the security of a system. Almost a year ago I blogged about the libsmpeg0 library which is listed as requiring an executable stack [1]. I submitted a two-line patch which fixes the problem while making no code changes (the patch gives the same result as running “execstack -c” on the resulting shared object).

My previous post documents the results of the problem when running SE Linux (a process is not permitted to run and an AVC message is logged). Some people might incorrectly think that this is merely a SE Linux functionality issue.

The program paxtest (which is in Debian but is i386 only) tests for a variety of kernel security features in terms of memory management. To demonstrate the problem that is caused by this issue I ran the commands “paxtest kiddie” and “LD_PRELOAD=/usr/lib/libsmpeg-0.4.so.0 paxtest kiddie“. The difference is that the test named “Executable stack” returns a result of Vulnerable when the object is loaded.

This means for example that attacks which rely on an executable stack will be permitted if the libsmpeg-0.4.so.0 shared object is loaded. So for example a program that loads the library and which takes data from the Internet (EG FreeCiv in network mode) will become vulnerable to attacks which rely on an executable stack because of this bug!

My Etch SE Linux repository has had a libsmpeg0 package which fixes this bug on i386 for almost a year [2] (the AMD64 packages are more recent). I have now added packages to fix this bug to my Lenny SE Linux repository [3]. I have also volunteered to NMU the package for Lenny. It seems that it would be rather embarrassing for everyone concerned systems were vulnerable to attack because of a two-line patch not being applied for almost a year.

I expect that the Release Team will be very accepting of package updates for Lenny which have patches to address this issue. A patch that has one line per assembler file (in the worst-case) to mark the object code is very easy to review. The results of the patch can be tested easily, and failure to have such a patch opens potential security holes. Package maintainers who can’t fix the assembly code can always run “execstack -c” in the build scripts to give the same result.

Lintian performs checks for executable stacks and the results are archived here [4]. There are currently 36 packages which contain binaries listed as needing executable stacks, I would be surprised if more than 6 of them actually contain shared objects that need an executable stack. If you use a package that is on that list then please test whether an executable stack is required by running “execstack -c” on the shared object and see if it still works. If a test of most of the high-level operations of the program in question can be completed successfully without an executable stack then it’s a strong indication that it’s not needed. Note that execstack is in the prelink package. I am happy to help with writing the patches to the packages and using my repositories to distribute the packages, but am not going to do so unless I can work with someone who uses the program in question and can test it’s functions. As an example of such testing I played a game of Frozen Bubble to test out the libsmpeg0 patch.

Xen CPU use per Domain

The command “xm list” displays the number of seconds of CPU time used by each Xen domain. This makes it easy to compare the CPU use of the various domains if they were all started at the same time (usually system boot). But is not very helpful if they were started at different times.

I wrote a little Perl program to display the percentage of one CPU that has been used by each domain, here is a sample of the output:

Domain-0 uses 7.70% of one CPU
demo uses 0.06% of one CPU
lenny32 uses 2.07% of one CPU
unstable uses 0.30% of one CPU

Now the command “xm top” can give you the amount of CPU time used at any moment (which is very useful). But it’s also good to be able to see how much is being used over the course of some days of operation. For example if a domain is using the equivalent of 34% of one CPU over the course of a week (as one domain that I run is doing) then it makes sense to allocate more than one VCPU to it so that things don’t slow down at peak times or when cron jobs are running.

I believe that it’s best to limit the number of CPUs allocated to Xen domains. For example I am running a Xen server with 8 CPU cores. I could grant each domain access to 8 VCPUs, but then any domain could use all that CPU power if it ran wild. While if I give no domain more than 2 VCPUs then one domain could use all the CPU resources allocated to it without the other domains being impacted. I realise that there are scheduling algorithms in the Xen kernel that are designed to deal with such situations, but I believe that simply denying access to excessive resource use is more effective and reliable.

I have not filed a bug report requesting that my script be added to one of the Xen packages as I’m not sure which one it would belong in (and also it’s a bit of a hack). It’s licensed under the GPL so anyone who wants to use it can do what they want. Any distribution package maintainer who wants to include it in a Xen utilities package is welcome to do so. The code is below.
Continue reading Xen CPU use per Domain