etbe - Russell Coker

SecureCon Lecture

On Thursday at Secure Con [1] I gave a lecture about SE Linux that went according to plan, and they gave me a nice bottle of Penfolds Shiraz afterwards (thanks to the sponsors).

During my lecture I announced my plan to run the hands-on training session over the net. The idea is that the Debian and CentOS images from jailtime.org with minor modifications will be put online somewhere for anyone to download. Anyone can then run the images on their own Xen server, go through the exercises, and ask questions on IRC at the same time. If you are interested in such training then please indicate in a comment what times would be good for the IRC discussion. Note that I’m only available between 7AM and 10PM starts in time zone +1100 (that is 20:00 to 11:00UTC for the starting time), the finishing time would be two hours later – and it would be possible to do the training in multiple sessions.

One interesting thing was that at the end the moderator of the session offered a box of lollies to the first person who could tell him my user-name (which was included in ls output on one of the slides).

Afterwards I was in idle conversation with some delegates and the topic of the Mac Mini [2] machines came up. Those machines are smaller than the Cobalt Qube (that I have in the past lugged around for portable SE Linux demonstrations), quite powerful (1G of RAM with an 80G hard drive seems to be the minimum for buying new at the moment), and they have keyboard and video ports which is often more convenient than sys-admin by serial port. I am now patiently waiting for Intel-based Mac Mini’s to start selling cheaply on eBay. Such a machine with 1G of RAM would make a nice SE Linux demo machine, I could run at least 7 Xen DomU’s for different users! Of course a second-hand laptop would do just as well, but laptops seem to hold their value better than most other machines.

One thing that disappointed me was the small turn-out for the conference dinner. It seemed that as there was a gap in the program between the official end of the conference at 5PM and dinner at 6PM most people decided to go home. One thing to note for future events is that leaving gaps in this way is probably a bad idea. Maybe if they had said “drinks at the restaurant from 5PM and dinner at 6PM” then the turn-out would have been better.

SecureCon Tutorial

My SecureCon tutorial went quite badly today. After having network problems and having both the Xen servers crash for no apparent reason I had to give up and give an impromptu lecture.

The original plan had been to use two Xen servers which each had 15 instances and have the delegates go through a training program that involved installing SE Linux on Debian and CentOS and comparing the features of them for various tasks.

Instead I spent just over two hours talking about SE Linux without notes (the beamer didn’t like my laptop and the desktop it was connected to was locked). I did end up getting another desktop machine working later in the lecture to type some notes.

My plan now is to make all the files available for download, additionally make some instances available on one of my servers, and then run some training via IRC.

Xen for Training

I’m setting up a training environment based on Xen. The configuration will probably be of use to some people so I’m including it below the fold. Please let me know if you have any ideas for improvements.

The interface for the user has the following documentation:

sudo -u root xen-manage create centos|debian [permissive]
Create an image, the parameter debian or centos specifies which
distribution you want to use and the optional parameter permissive
specifies that you want to use Permissive mode (no SE Linux access controls
enforced).
Note that creating an image will leave you at it’s console. Press ^]
to escape from the console.
sudo -u root xen-manage list
Display the Xen formation on your DomU. Note that it doesn’t tell you whether
you are using Debian or CentOS, you have to access the console to do that.
sudo -u root xen-manage console
Access the console.
sudo -u root xen-manage destroy
Destroy your Xen image – if it’s crashed and you want to restart it.

Continue reading Xen for Training

Squid and SE Linux

Is Squid not returning some data you need on a SE Linux system?

The default configuration of the SE Linux policy for Squid only allows it to connect to a small number of ports which are used for web servers. For example ports http (80) and https (443) are labelled as http_port_t which permits serves such as Apache to bind to them and Squid to connect to them. But sometimes services run on non-standard ports and periodically new services are devised which use the HTTP protocol and thus you have Squid and Apache using new ports.

semanage port -a -t http_port_t -p tcp 11371

One example of such a port is hkp (11371) – the latest protocol for sending and receiving GPG/OpenPGP keys. Running the above command relabelled the TCP port 11371 in question as http_port_t and thus allowed everything to work.

setsebool -P squid_connect_any 1
An alternate option would be to run the above command to allow Squid to connect to any port.

I will suggest that the upstream policy be changed to make the default labelling of TCP port 11371 be http_port_t, but the same operations can be used for other ports.

Some people may claim that this makes things difficult for sys-admins. But the fact is that a well known port is a significant resource that you don’t want to permit any random user to access. Not only do the SE Linux port access controls prevent malice, but they also prevent system programs from accidentally using the wrong ports. A common example of accidental mis-use is the port 631 used for the IPP (Internet Printing Protocol – CUPS). When system programs need to use TCP source ports below 1024 they start at 1023 and work their way down, having such programs get down to 631 is not uncommon (there are some error conditions which result in ports being reserved for some minutes after use). In terms of malicious operations, it seems that the ports used by database servers such as MySQL and PostgreSQL would ideally be inaccessible to a Squid proxy, and services such as network backup should be inaccessible to everything other than the backup software.

Increasing Efficiency through Less Work

I have just read an interesting article titled Why Crunch Mode Doesn’t Work [1] which documents the research on efficiency vs amount of time spent working (and by inference amount of time spent on leisure activities and sleep). It shows that a 40 hour working week was chosen by people who run factories (such as Henry Ford) not due to being nice for the workers but due to the costs of inefficient work practices and errors that damage products and equipment.

Now these results can only be an indication of what works best by today’s standards. The military research is good but only military organisations get to control workers to that degree (few organisations try to control how much sleep their workers get or are even legally permitted to do so), companies can only give their employees appropriate amounts of spare time to get enough sleep and hope for the best.

Much of the research dates from 80+ years ago. I suspect that modern living conditions where every house has electric lights and entertainment devices such as a TV to encourage staying awake longer during the night will change things, as would ubiquitous personal transport by car. It could be that for modern factory workers the optimum amount of work is not 40 hours a week, it could be as little as 30 or as much as 50 (at a guess).

Also the type of work being done certainly changes things. The article notes that mental tasks are affected more than physical tasks by lack of sleep (in terms of the consequences of being over-tired), but no mention is made about whether the optimum working hours change. If the optimum amount of work in a factory is 40 hours per week might the optimum for a highly intellectual task such as computer programming be less, perhaps 35 or 30?

The next factor is the issue of team-work. In an assembly-line it’s impossible to have one person finish work early while the rest keep working, so the limit will be based on the worker who can handle the least hours. Determining which individuals will work more slowly when they work longer hours is possible (but it would be illegal to refuse to hire such people in many jurisdictions) and determining which individuals might be more likely to cause industrial accidents may be impossible. So it seems to me that the potential for each employee to work their optimal hours is much greater in the computer industry than in most sectors. I have heard a single anecdote of an employee who determined that their best efficiency came from 5 hours work a day and arranged with their manager to work 25 hours a week, apart from that I have not heard any reports of anyone trying to tailor the working hours to the worker.

Some obvious differences in capacity for working long hours without losing productivity seem related to age and general health, obligations outside work (EG looking after children or sick relatives), and enjoyment of work (the greater the amount of work time that can be regarded as “fun” the less requirement there would be for recreation time outside work). It seems likely to me that parts of the computer industry that are closely related to free software development could have longer hours worked due to the overlap between recreation and paid work.

If the amount of time spent working was to vary according to the capacity of each worker then the company structures for management and pay would need to change. Probably the first step towards this would be to try to pay employees according to the amount of work that they do, one problem with this is the fact that managers are traditionally considered to be superior to workers and therefore inherently worthy of more pay. As long as the pay of engineers is restricted to less than the pay of middle-managers the range between the lowest and highest salaries among programmers is going to be a factor of at most five or six, while the productivity difference between the least and most skilled programmers will be a factor of 20 for some boring work and more than 10,000 for more challenging work (assuming that the junior programmer can even understand the task). I don’t expect that a skillful programmer will get a salary of $10,000,000 any time soon (even though it would be a bargain compared to the number of junior programmers needed to do the same work), but a salary in excess of $250,000 would be reasonable.

If pay was based on the quality and quantity of work done (which as the article mentions is difficult to assess) then workers would have an incentive to do what is necessary to improve their work – and with some guidance from HR could adjust their working hours accordingly.

Another factor that needs to be considered is that ideally the number of working hours would vary according to the life situation of the worker. Having a child probably decreases the work capacity for the next 8 years or so.

These are just some ideas, please read the article for the background research. I’m going to bed now. ;)

[1] https://tinyurl.com/y66tyev7

Load Average

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep).

There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number.

What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time – although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5.

The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file – ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time.

For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task).

The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux|grep D” (to get a list of D state processes – processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states.

Cpu(s): 15.0%us, 35.1%sy, 0.0%ni, 49.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7331 rjc 25 0 2868 640 312 R 100 0.0 0:21.57 gzip

Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core – a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle).

Cpu(s): 1.3%us, 3.2%sy, 0.0%ni, 50.7%id, 44.4%wa, 0.0%hi, 0.3%si, 0.0%st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7425 rjc 17 0 4036 872 588 R 4 0.1 0:00.20 find

Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time.

Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out.

Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average.

His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

Update: Jon Oxer really liked this post.

Halloween

Yesterday I received an unsigned notice in the mail from some residents of an area comprising my street and an adjacent one. They advised me that their children were going to do the Halloween thing and that if I wanted to be involved I should leave my porch light on. This is a really good idea, people who like that sort of thing can leave their light on and give lollies to children (whatever happened to “don’t take lollies from strangers”?). People who don’t like pagan festivals can leave their porch light off and not be bothered.

One thing that occurred to me is that the anonymous flyer might have been posted by someone who doesn’t like Halloween to provide a way for themselves and like-minded people to opt-out of it. I’m sure that everyone told their children not to knock on a door unless there is a porch light on.

It’s almost 10PM and no-one has rung my doorbell, it seems to have worked.

SecureCon 2007

I am running a tutorial and giving a talk about SE Linux at SecureCon 2007 [1].

The tutorial will go for 3 hours on Wednesday the 7th of November and will cover using SE Linux in CentOS 5 and Debian Etch, it will be a hands-on tutorial where every delegate gets ssh access to their own Xen DomU.

The lecture is on Thursday the 8th of November and will be a 45 minute talk with an overview of SE Linux. It will be similar to my speech at the AUUG conference [2] but probably cover more of the features. The AUUG talk was driven by questions from the audience to spend a lot of time justifying SE Linux design decisions which took time away from other materiel. This wasn’t inherently a problem (I provided the information the audience seemed to want and everyone seemed happy), but I would like to cover more of the features and new developments.

New SE Linux Play Machine Online

After over a year I have finally got a SE Linux Play Machine online again.

The details for logging in are at this link [1]. I’ve created T-shirt and mug designs with the login details too, they are on cafepress.com LINK [2]. For fun wear such a shirt to a conference (or even when shopping at your local electronics store. ;)

Xen and Security

I have previously posted about the difference between using a chroot and using SE Linux [1].

Theo de Raadt claims that virtualisation does not provide security benefits [2] based on the idea that the Xen hypervisor may have security related bugs.

From my understanding of Xen a successful exploit of a Xen system with a Dom0 that is strictly used for running the DomU’s would usually start by gaining local root on one of the DomU instances. From there it is possible to launch an attack on the Xen Dom0. One example of this is the recent Xen exploit (CVE-2007-4993) [3] where hostile data in a grub.conf in a DomU could be used to execute privileged commands in the Dom0. Another possibility would be to gain root access to a DomU and then exploit a bug in the Xen API to take over the hypervisor (I am not aware of an example of this being implemented). A final possibility is available when using QEMU code to provide virtual hardware where an attacker could exploit QEMU bugs, an example of this is CVE-2007-0998 where a local user in a guest VM could read arbitrary files in the host [4] – it’s not clear from the advisory what level of access is required to exploit it (DomU-user, DomU-root, or remote VNC access). VNC is different from other virtual hardware in that the sys-admin of the virtual machine (who might be untrusted) needs to access it. Virtual block devices etc are only accessed by the DomU and Xen manages the back-end.

The best reference in regard to these issues seems to be Tavis Ormandy’s paper about hostile virtualised environments [5]. Tavis found some vulnerabilities in the QEMU hardware emulation, and as QEMU code is used for a fully virtualised Xen installation it seems likely that Xen has some vulnerabilities in this regard. I think that it is generally recommended that for best security you don’t run fully virtualised systems.

The remote-console type management tools are another potential avenue of attack for virtualised servers in the case where multiple users run virtual machines on the same host (hardware). I don’t think that this is an inherent weakness of virtualisation systems. When security is most important you have one sys-admin running all virtual machines – which incidentally seems to be the case for most implementations of Xen at the moment (although for management not security reasons). In ISP hosting type environments I doubt that a remote console system based on managing Xen DomU’s is going to be inherently any less secure than a typical remote console system for managing multiple discrete computers or blades.

I have just scanned the Xen hypervisor source, the file include/asm-x86/hypercall.h has 18 entries for AMD64 and 17 for i386 while include/xen/hypercall.h has 18 entries. So it seems that there are 35 or 36 entry points to call the hypervisor, compared to 296 system calls on the i386 version of Linux (which includes the sys_socketcall system call which expands to many system calls). This seems to be one clear indication that the Linux kernel is inherently more complex (and therefore likely to have a higher incidence of security flaws) than the Xen hypervisor.

Theo’s main claim seems to be that Xen is written by people who aren’t OpenBSD developers and who therefore aren’t able to write secure code. While I don’t agree with his strong position I have to note the fact that OpenBSD seems to have a better security history than any other multi-user kernel for which data is available. But consider a system running Xen with Linux in Dom0 and multiple para-virtualised OpenBSD DomU’s. If the Linux Dom0 has OpenSSH as the only service being run then the risk of compromise would be from OpenSSH, IP based bugs in the Linux kernel (either through the IP address used for SSH connections or for routing/bridging to the OpenBSD instances), and from someone who has cracked root on one of the OpenBSD instances and is attacking the hypervisor directly.

Given that OpenSSH comes from the OpenBSD project it seems that the above scenario would only add the additional risk of an IP based Linux kernel attack. While a root compromise of an OpenBSD instance (consider that a typical OpenBSD system will run a lot of software that doesn’t come from the OpenBSD project – much of which won’t have a great security history) would only lose that instance unless the attacker can also exploit the hypervisor (which would be a much more difficult task than merely cracking some random daemon running as root that the sys-admin is forced to install). Is the benefit of having only one instance of OpenBSD cracked due to a bad daemon enough to outweigh the risk of a Linux IP stack?

I’m sure that the OpenBSD people would consider that a better option would be OpenBSD in the Dom0 and in the DomU. In which case the risk of damage from a root compromise due to one badly written daemon that didn’t come from OpenBSD is limited to a single DomU unless the attacker also compromises the hypervisor. When working as a sys-admin I have been forced by management to install some daemons as root which were great risks to the security of the system, if I had the ability to install them in separate DomU’s I would have been able to significantly improve the security of the system.

Another flaw in Theo’s position is that he seems to consider running a virtual machine as the replacement of multiple machines – which would be an obvious decrease in security. However in many cases the situation is that no more or less hardware is purchased, it is just used differently. If instead of a single server running several complex applications you have a Xen server running multiple DomU’s which each have a single application then things become much simpler and more secure. Upgrades can be performed on one DomU at a time which decreases the scope of failure (which often means that you only need one business unit to sign-off on the upgrade) and upgrades can be performed on an LVM snapshot (and rolled back with ease if they don’t succeed). A major problem with computer security is when managers fear problems caused by upgrades and prohibit their staff from applying security fixes. This combined with the fact that on a multiple DomU installation one application can be compromised without immediate loss of the others (which run in different DomU’s and require further effort by the attacker for a Xen compromise) provides a significant security benefit.

It would be nice for security if every application could run on separate hardware, but even with blades this is not economically viable – not even for the biggest companies.

I have converted several installations from a single overloaded and badly managed server to a Xen installation with multiple DomU’s. In all cases the DomU’s were easier to upgrade (and were upgraded more often) and the different applications and users were more isolated.

Finally there is the possibility of using virtualisation to monitor the integrity of the system, Bill Broadley’s presentation from the 2007 IT Security Symposium [6] provides some interesting ideas about what can be done. It seems that having a single OpenBSD DomU running under a hypervisor (maybe Xen audited by the OpenBSD people) with an OpenBSD Dom0 would offer some significant benefits over a single OpenBSD instance.

etbe – Russell Coker

Archives

Categories

SecureCon Lecture

SecureCon Tutorial

Xen for Training

Squid and SE Linux

Increasing Efficiency through Less Work

Load Average

Halloween

SecureCon 2007

New SE Linux Play Machine Online

Xen and Security

Archives

Email and RSS

Archives

Categories

Tags

Archives

Email and RSS