Software vs Hardware RAID

November 16, 2007 | etbe

Should you use software or hardware RAID? Many people claim that Hardware RAID is needed for performance (which can be true) but then claim that it’s because of the CPU use of the RAID calculations.

Here is the data logged by the Linux kernel then the RAID-5 and RAID-6 drivers are loaded on a 1GHz Pentium-3 system:

raid5: automatically using best checksumming function: pIII_sse
pIII_sse : 2044.000 MB/sec
raid5: using function: pIII_sse (2044.000 MB/sec)
raid6: int32x1 269 MB/s
raid6: int32x2 316 MB/s
raid6: int32x4 308 MB/s
raid6: int32x8 281 MB/s
raid6: mmxx1 914 MB/s
raid6: mmxx2 1000 MB/s
raid6: sse1x1 800 MB/s
raid6: sse1x2 1046 MB/s
raid6: using algorithm sse1x2 (1046 MB/s)

There are few P3 systems that have enough IO capacity to support anywhere near 2000MB/s of disk IO and modern systems give even better CPU performance.

The fastest disks available can sustain about 80MB/s when performing contiguous disk IO (which incidentally is a fairly rare operation). So if you had ten fast disks performing contiguous IO then you might be using 800MB/s of disk IO bandwidth, but that would hardly stretch your CPU performance. It’s obvious that CPU performance of the XOR calculations for RAID-5 (and the slightly more complex calculations for RAID-6) is not a bottleneck.

Hardware RAID-5 often significantly outperforms software RAID-5 (in fact it should always outperform software RAID-5) even though in almost every case the RAID processor has significantly less CPU power than the main CPU. The benefit for hardware RAID-5 is in caching. A standard feature in a hardware RAID controller is a write-back disk cache in non-volatile RAM (RAM that has a battery backup and can typically keep it’s data for more than 24 hours without power). All RAID levels benefit from this but RAID-5 and RAID-6 gain particular benefits. In RAID-5 a small write (less than the stripe size) requires either that all the blocks other than the ones to be written are read or that the original content of the block to be written and the parity block are read – in either case writing less than a full stripe to a RAID-5 requires some reads. If the write-back cache can store the data for long enough that a second write is performed to the same stripe (EG to files being created in the same Inode block) then performance may double.

There is one situation where software RAID will give better performance (often significantly better performance), that is for low-end hardware RAID devices. I suspect that some hardware RAID vendors deliberately cripple the performance of low-end RAID devices (by using an extremely under-powered CPU among other things) to drive sales of the more expensive devices. In 2001 I benchmarked one hardware RAID controller as being able to only sustain 10MB/s for contiguous read and write operations (software RAID on lesser hardware would deliver 100MB/s or more). But for random synchronous writes the performance was great and that’s what mattered for the application in question.

Also there are reliability issues related to write-back caching. In a well designed system an update of an entire RAID-5 stripe (one block to each disk including the parity block) will first be performed to the cache and then the cache will be written back. If the power fails while the write is in progress then it will be attempted again when power is restored thus ensuring that all disks have the same data. With any RAID implementation without such a NVRAM cache a write to the entire stripe could be partially successful. This means that the parity block would not match the data! In such a situation the machine would probably work well (fsck would ensure that the filesystem state was consistent) until a disk failed. When the RAID-5 recovery procedure is used after a disk is failed it uses the parity block to re-generate the missing data, but if the parity doesn’t match then the re-generated data will be different. A disk failure may happen while the machine is online and this could potentially result in filesystem and/or database meta-data changing on a running system – this is a bad situation that most filesystems and databases will not handle well.

A further benefit of a well designed NVRAM cache is that it can be used on multiple systems. For their servers HP makes some guarantees about which replacement machines will accept the NVRAM module. So if you have a HP server running RAID-5 with an NVRAM cache then you could have the entire motherboard die, have HP support provide a replacement server, then when the replacement machine is booted with the old hard drives and NVRAM module installed the data in the write-back cache will be written! This is a significant feature for improving reliability in bad corner cases. NB I’m not saying that HP is better than all other RAID vendors in this regard, merely that I know what HP equipment will do and don’t know about the rest.

It would be good if there was a commodity standard for NVRAM on a PC motherboard. Perhaps a standard socket design that Intel could specify and that every motherboard manufacturer would eventually support. Then to implement such things on a typical PC all that would be required would be the NVRAM module, which while still being expensive would be significantly cheaper than current prices due to the increase in volume. If there was a significant quantity of PCs with such NVRAM (or which could be upgraded to it without excessive cost) then there would be an incentive for people to modify the Linux sotware RAID code to use it and thus give benefits for performance and reliability. Then it could be possible to install a NVRAM module and drives in a replacement server with Linux software RAID and have the data integrity preserved. But unless/until such things happen write-back caching that preserves the data integrity requires hardware RAID.

Another limitation of Linux software RAID is expanding RAID groups. A HP server that I work on had two disks in a RAID-1 array, one of my colleagues added an extra disk and made it a RAID-5, the hardware RAID device moved the data around as appropriate while the machine was running and the disk space was expanded without any down-time. Some similar things can be done with Linux, for example here is documentation on converting RAID-1 to RAID-5 with Linux software RAID [1]. But that conversion operation requires some down-time and is not something that’s officially supported, while converting RAID-1 to RAID-5 with HP hardware RAID is a standard supported feature.

[1] http://scott.wallace.sh/node/1521

More About Living in Hotels

November 14, 2007 | etbe

In the past I have spent about 18 months living in hotels with a couple of months of breaks in between. I have previously written about it in terms of living in London hotels [1], but I have been asked for more generic advice.

Firstly the amount of possessions that you may have when living in hotels is seriously limited. For ease of travel you want to restrict yourself to one suitcase for checked luggage and one for carry-on luggage. Hotels often have short-term storage space for possessions of guests, so having a second suitcase of items that are not worth stealing (clothes and books) may be an option. But consumer electronics devices other than a single laptop computer are not an option.

I read an interesting blog post on ZenHabits.net titled Minimalist Fun: The 100 Things Challenge [2] which advocates counting and limiting the number of possessions you own. When living in hotels if I considered my books as one collection and my clothes as another (having never been interested in trendy clothes they count as utilitarian items for work or leisure not objects that I seek to own) and as my mobile phone was a tool for work and my computer gear was strictly limited to items that were needed for work (and thus “tools”) my only possessions were a digital camera and some bottles of liquor! The lack of ability to accumulate possessions may be considered as an advantage or a disadvantage depending on what your aims are.

If you are moving to another country for work there are three ways of doing it. The easiest is to be a permanent employee of a company that assigns you to work there – in which case they will probably pay to transport your stuff when you buy or rent a house. If you are a looking for new employment (either contract or permanent) in another country then you can either find the work before moving or after arriving there. Finding work before arriving in the country is difficult and generally only works for short-term contracts. So it’s most likely that you will either be looking for work immediately after arriving or after a short contract. In either case having better mobility increases your employment options – why restrict yourself to one city or region when you can choose from all jobs in an entire country or (in the case of the EU) half of a continent! The career benefits of being able to accept any job anywhere in the world at short notice are significant!

There are situations where an employer will pay hotel bills. One example was when I was working for a London based company and they assigned us to work at the other side of London. My colleagues complained and the company paid for hotel bills for everyone Sunday night to Thursday night inclusive as well as an extra hour of pay per day as compensation for the inconvenience. For me of course one hotel was as good as another so it just meant that my employer was covering 5/7 of my living expenses. Then I had a meeting with the hotel manager and pointed out that having me check out every Friday would be bad for them as the hotel was mostly empty on the weekend and suggested that they make me a deal for the other two days – I ended up paying something like one night of hotel fees per week! If I had rented an apartment I would have still been paying the full rent (which while less than 30 days hotel fees per month would have been considerably more than 4 or 5 days of hotel fees per month).

If you live in a hotel then there is always some sort of deal that can be arranged. Apart from certain busy days (such as around the Christmas and new-year time) they always want long-term guests and will be willing to reduce the price, give free dinner or drinks from the bar, etc.

The cost of living in a hotel at times such as Christmas may be as much as five times the regular rate. That is a further incentive to visit friends or relatives at Christmas. If you can’t visit your family (which may be difficult if they live on the other side of the world) then finding a friend who has a spare room might be an option.

Restorecon Equivalent for Unix Permissions

November 13, 2007 | etbe

SE Linux has a utility named restorecon to set (or reset) the security context. This is useful for many reasons, corrupted filesystems, users removing files or changing the context in inappropriate ways, and for re-creating files from tar files or backup programs that don’t restore SE Linux contexts. It can also be used to report the files that have different contexts to that which would be set by restorecon to verify the contexts of files.

Restorecon determines the context from two sources of data, one is the policy that came with the system (including any policy modules from other sources which were loaded) and the other is the local file contexts that were created by semanage.

It’s a pity that there doesn’t seem to be an equivalent program for Unix permissions. rpm has a -V option to verify the files from a package and dpkg doesn’t seem to have an option to perform a similar operation (/var/lib/dpkg/info/* doesn’t seem to have the necessary data). But even on an RPM based system this isn’t possible because there is no way to add local files into the list.

I would like to be able to specify that an RPM system should have root:root as the owner and permission mode 0755 for all files matching /usr/local/bin/* and use a single command to check the RPM database as well as this extra data for the permissions of all files.

Does anyone know of any work in this area?

I’m going to file Debian and Fedora bug reports about this, but I would appreciate any comments first.

Update:

Here is an example of how this feature works in rpm:
# rpm -Vv nash
…….. /sbin/nash
…….. d /usr/share/man/man8/nash.8.gz
# chmod 700 /sbin/nash
# rpm -Vv nash
.M…… /sbin/nash
…….. d /usr/share/man/man8/nash.8.gz

The “M” character indicates that the permission mode of the file does not match the RPM. There is no way to automatically correct it (AFAIK) but at least we know that something changed. With Debian AFAIK it’s only possible to verify file checksums not the permission.

SecureCon Lecture

November 10, 2007 | etbe

On Thursday at Secure Con [1] I gave a lecture about SE Linux that went according to plan, and they gave me a nice bottle of Penfolds Shiraz afterwards (thanks to the sponsors).

During my lecture I announced my plan to run the hands-on training session over the net. The idea is that the Debian and CentOS images from jailtime.org with minor modifications will be put online somewhere for anyone to download. Anyone can then run the images on their own Xen server, go through the exercises, and ask questions on IRC at the same time. If you are interested in such training then please indicate in a comment what times would be good for the IRC discussion. Note that I’m only available between 7AM and 10PM starts in time zone +1100 (that is 20:00 to 11:00UTC for the starting time), the finishing time would be two hours later – and it would be possible to do the training in multiple sessions.

One interesting thing was that at the end the moderator of the session offered a box of lollies to the first person who could tell him my user-name (which was included in ls output on one of the slides).

Afterwards I was in idle conversation with some delegates and the topic of the Mac Mini [2] machines came up. Those machines are smaller than the Cobalt Qube (that I have in the past lugged around for portable SE Linux demonstrations), quite powerful (1G of RAM with an 80G hard drive seems to be the minimum for buying new at the moment), and they have keyboard and video ports which is often more convenient than sys-admin by serial port. I am now patiently waiting for Intel-based Mac Mini’s to start selling cheaply on eBay. Such a machine with 1G of RAM would make a nice SE Linux demo machine, I could run at least 7 Xen DomU’s for different users! Of course a second-hand laptop would do just as well, but laptops seem to hold their value better than most other machines.

One thing that disappointed me was the small turn-out for the conference dinner. It seemed that as there was a gap in the program between the official end of the conference at 5PM and dinner at 6PM most people decided to go home. One thing to note for future events is that leaving gaps in this way is probably a bad idea. Maybe if they had said “drinks at the restaurant from 5PM and dinner at 6PM” then the turn-out would have been better.

SecureCon Tutorial

November 7, 2007 | etbe

My SecureCon tutorial went quite badly today. After having network problems and having both the Xen servers crash for no apparent reason I had to give up and give an impromptu lecture.

The original plan had been to use two Xen servers which each had 15 instances and have the delegates go through a training program that involved installing SE Linux on Debian and CentOS and comparing the features of them for various tasks.

Instead I spent just over two hours talking about SE Linux without notes (the beamer didn’t like my laptop and the desktop it was connected to was locked). I did end up getting another desktop machine working later in the lecture to type some notes.

My plan now is to make all the files available for download, additionally make some instances available on one of my servers, and then run some training via IRC.

Xen for Training

November 7, 2007 | etbe

I’m setting up a training environment based on Xen. The configuration will probably be of use to some people so I’m including it below the fold. Please let me know if you have any ideas for improvements.

The interface for the user has the following documentation:

sudo -u root xen-manage create centos|debian [permissive]
Create an image, the parameter debian or centos specifies which
distribution you want to use and the optional parameter permissive
specifies that you want to use Permissive mode (no SE Linux access controls
enforced).
Note that creating an image will leave you at it’s console. Press ^]
to escape from the console.
sudo -u root xen-manage list
Display the Xen formation on your DomU. Note that it doesn’t tell you whether
you are using Debian or CentOS, you have to access the console to do that.
sudo -u root xen-manage console
Access the console.
sudo -u root xen-manage destroy
Destroy your Xen image – if it’s crashed and you want to restart it.

Continue reading →

Squid and SE Linux

November 6, 2007 | etbe

Is Squid not returning some data you need on a SE Linux system?

The default configuration of the SE Linux policy for Squid only allows it to connect to a small number of ports which are used for web servers. For example ports http (80) and https (443) are labelled as http_port_t which permits serves such as Apache to bind to them and Squid to connect to them. But sometimes services run on non-standard ports and periodically new services are devised which use the HTTP protocol and thus you have Squid and Apache using new ports.

semanage port -a -t http_port_t -p tcp 11371

One example of such a port is hkp (11371) – the latest protocol for sending and receiving GPG/OpenPGP keys. Running the above command relabelled the TCP port 11371 in question as http_port_t and thus allowed everything to work.

setsebool -P squid_connect_any 1
An alternate option would be to run the above command to allow Squid to connect to any port.

I will suggest that the upstream policy be changed to make the default labelling of TCP port 11371 be http_port_t, but the same operations can be used for other ports.

Some people may claim that this makes things difficult for sys-admins. But the fact is that a well known port is a significant resource that you don’t want to permit any random user to access. Not only do the SE Linux port access controls prevent malice, but they also prevent system programs from accidentally using the wrong ports. A common example of accidental mis-use is the port 631 used for the IPP (Internet Printing Protocol – CUPS). When system programs need to use TCP source ports below 1024 they start at 1023 and work their way down, having such programs get down to 631 is not uncommon (there are some error conditions which result in ports being reserved for some minutes after use). In terms of malicious operations, it seems that the ports used by database servers such as MySQL and PostgreSQL would ideally be inaccessible to a Squid proxy, and services such as network backup should be inaccessible to everything other than the backup software.

Increasing Efficiency through Less Work

November 2, 2007 | etbe

I have just read an interesting article titled Why Crunch Mode Doesn’t Work [1] which documents the research on efficiency vs amount of time spent working (and by inference amount of time spent on leisure activities and sleep). It shows that a 40 hour working week was chosen by people who run factories (such as Henry Ford) not due to being nice for the workers but due to the costs of inefficient work practices and errors that damage products and equipment.

Now these results can only be an indication of what works best by today’s standards. The military research is good but only military organisations get to control workers to that degree (few organisations try to control how much sleep their workers get or are even legally permitted to do so), companies can only give their employees appropriate amounts of spare time to get enough sleep and hope for the best.

Much of the research dates from 80+ years ago. I suspect that modern living conditions where every house has electric lights and entertainment devices such as a TV to encourage staying awake longer during the night will change things, as would ubiquitous personal transport by car. It could be that for modern factory workers the optimum amount of work is not 40 hours a week, it could be as little as 30 or as much as 50 (at a guess).

Also the type of work being done certainly changes things. The article notes that mental tasks are affected more than physical tasks by lack of sleep (in terms of the consequences of being over-tired), but no mention is made about whether the optimum working hours change. If the optimum amount of work in a factory is 40 hours per week might the optimum for a highly intellectual task such as computer programming be less, perhaps 35 or 30?

The next factor is the issue of team-work. In an assembly-line it’s impossible to have one person finish work early while the rest keep working, so the limit will be based on the worker who can handle the least hours. Determining which individuals will work more slowly when they work longer hours is possible (but it would be illegal to refuse to hire such people in many jurisdictions) and determining which individuals might be more likely to cause industrial accidents may be impossible. So it seems to me that the potential for each employee to work their optimal hours is much greater in the computer industry than in most sectors. I have heard a single anecdote of an employee who determined that their best efficiency came from 5 hours work a day and arranged with their manager to work 25 hours a week, apart from that I have not heard any reports of anyone trying to tailor the working hours to the worker.

Some obvious differences in capacity for working long hours without losing productivity seem related to age and general health, obligations outside work (EG looking after children or sick relatives), and enjoyment of work (the greater the amount of work time that can be regarded as “fun” the less requirement there would be for recreation time outside work). It seems likely to me that parts of the computer industry that are closely related to free software development could have longer hours worked due to the overlap between recreation and paid work.

If the amount of time spent working was to vary according to the capacity of each worker then the company structures for management and pay would need to change. Probably the first step towards this would be to try to pay employees according to the amount of work that they do, one problem with this is the fact that managers are traditionally considered to be superior to workers and therefore inherently worthy of more pay. As long as the pay of engineers is restricted to less than the pay of middle-managers the range between the lowest and highest salaries among programmers is going to be a factor of at most five or six, while the productivity difference between the least and most skilled programmers will be a factor of 20 for some boring work and more than 10,000 for more challenging work (assuming that the junior programmer can even understand the task). I don’t expect that a skillful programmer will get a salary of $10,000,000 any time soon (even though it would be a bargain compared to the number of junior programmers needed to do the same work), but a salary in excess of $250,000 would be reasonable.

If pay was based on the quality and quantity of work done (which as the article mentions is difficult to assess) then workers would have an incentive to do what is necessary to improve their work – and with some guidance from HR could adjust their working hours accordingly.

Another factor that needs to be considered is that ideally the number of working hours would vary according to the life situation of the worker. Having a child probably decreases the work capacity for the next 8 years or so.

These are just some ideas, please read the article for the background research. I’m going to bed now. ;)

[1] https://tinyurl.com/y66tyev7

Load Average

November 1, 2007 | etbe

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep).

There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number.

What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time – although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5.

The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file – ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time.

For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task).

The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux|grep D” (to get a list of D state processes – processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states.

Cpu(s): 15.0%us, 35.1%sy, 0.0%ni, 49.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7331 rjc 25 0 2868 640 312 R 100 0.0 0:21.57 gzip

Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core – a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle).

Cpu(s): 1.3%us, 3.2%sy, 0.0%ni, 50.7%id, 44.4%wa, 0.0%hi, 0.3%si, 0.0%st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7425 rjc 17 0 4036 872 588 R 4 0.1 0:00.20 find

Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time.

Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out.

Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average.

His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

Update: Jon Oxer really liked this post.

Halloween

October 31, 2007 | etbe

Yesterday I received an unsigned notice in the mail from some residents of an area comprising my street and an adjacent one. They advised me that their children were going to do the Halloween thing and that if I wanted to be involved I should leave my porch light on. This is a really good idea, people who like that sort of thing can leave their light on and give lollies to children (whatever happened to “don’t take lollies from strangers”?). People who don’t like pagan festivals can leave their porch light off and not be bothered.

One thing that occurred to me is that the anonymous flyer might have been posted by someone who doesn’t like Halloween to provide a way for themselves and like-minded people to opt-out of it. I’m sure that everyone told their children not to knock on a door unless there is a porch light on.

It’s almost 10PM and no-one has rung my doorbell, it seems to have worked.

etbe – Russell Coker

Linux, politics, and other interesting things