Below is a GNUPlot graph of ZCAV output from a 250G SATA disk. The disk has something wrong with it (other disks of the same brand in machines of the same configuration give more regular graphs). The expected graph will start high on the Y scale (MB/s) and steadily drop as the reads go to shorter tracks near the spindle. The ZCAV results page has some examples of the expected results.
If you have any ideas as to what might cause this performance result then please make a comment on this blog post.
Update: This problem is solved, I’ve written a new post explaining the answer. It’s a pity that an NDA delayed my publication of this information.
This paper by Rodney Van Meter about ZCAV (Zoned Constant Angular Velocity) in hard drives is very interesting. It predates my work by about four years and includes some interesting methods of collecting data that I never considered.
One interesting thing is that apparently on some SCSI drives you can get the drive to tell you where the zones are. If I get enough spare time I would like to repeat such tests and see how the data returned by disks compares to benchmark results.
It’s also interesting to note that Rodney’s paper shows a fairly linear drop of performance on higher sector numbers (while he notes that it would be expected to fall off more quickly at higher sector numbers). One of my recent tests with a 300G disk showed the greater than linear performance drop (see my ZCAV results page for more details). It might require modern large disks to show this performance characteristic.
I also found it very interesting to see that a modified version of Bonnie was used for some of the tests and that it gave consistent results! I assumed that any filesystem based tests of ZCAV performance would introduce unreasonable amounts of variance into my tests and instead wrote my ZCAV test program to directly read the disk and measure performance.
It’s times like this that I wish for a “groundhog day” so that I could spend a year doing nothing but reading technical papers.
Today I have released a significant new version of my mail server benchmark Postal! The list of changes is below:
- Added new program bhm to listen on port 25 and send mail to /dev/null. This allows testing mail relay systems.
- Fixed a minor bug in reporting when compiled without SSL.
- Made postal write the date header in correct RFC2822 format.
- Removed the name-expansion feature, it confused many people and is not needed now that desktop machines typically have 1G of RAM. Now postal and rabid can have the same user-list file.
- Moved postal-list into the bin directory.
- Changed the thread stack size to 32K (used to be the default of 10M) to save virtual memory size (not that this makes much difference to anything other than the maximum number of threads on i386).
- Added a minimum message size option to Postal (now you can use fixed sizes).
- Added a Postal option to specify a list of sender addresses separately to the list of recipient addresses.
- Removed some unnecessary error messages.
- Handle EINTR to allow ^Z and “bg” from the command line. I probably don’t handle all cases, but now that I agree that failure to handle ^Z is an error I expect bug reports.
- Made the test programs display output on the minute, previously they displayed once per minute (EG 11:10:35) while now it will be 11:10:00. This also means that the first minute reported will have something less than 60 seconds of data – this does not matter as a mail server takes longer than that to get up to speed.
- Added support for GNUTLS and made the Debian package build with it. Note that BHM doesn’t yet work correctly with TLS.
- Made the programs exit cleanly.
Thanks to Inumbers for sponsoring the development of Postal.
I presented a paper on mail server performance at OSDC 2006 that was based on the now-released version of Postal.
I’ve been replying to a number of email messages in my Postal backlog, some dating back to 2001. Some of the people had changed email address during that time so I’ll answer their questions in my blog instead.
/usr/include/openssl/kssl.h:72:18: krb5.h: No such file or directory
In file included from /usr/include/openssl/ssl.h:179
One problem reported by a couple of people is having the above error when compiling on an older Red Hat release (RHL or RHEL3). Running ./configure –disable-ssl should work-around that problem (at the cost
of losing SSL support). As RHEL3 is still in support I plan to fix this bug eventually.
There was a question about how to get detailed information on what Postal does. All three programs support the options -z and -Z to log details of what they do. This isn’t convenient if you only want a small portion of the data but can be used to obtain any information you desire.
One user reported that options such as -p5 were not accepted. Apparently their system had a broken implementation of the getopt(3) library call used to parse command-line parameters.
Yesterday I gave a presentation at OSDC in Melbourne about my Postal mail server benchmark suite. The paper was about my new benchmark program BHM for testing the performance of mail relay systems and some of the things I learned by running it. I will put the paper on my Postal site in the near future and also I’ll release a new version of Postal with the code in question very soon.
Today at OSDC I gave a 5 minute talk on 5 things that need to be improved in the security of Linux distributions.
- The fact that unprivileged programs often inherit the controlling tty of privileged programs which permits them to use the TIOCSTI ioctl to insert characters in the keyboard buffer. I noted that with runuser and a subtle change to su things have been significantly improved in this regard in Fedora, but other distributions need work (and Fedora can go further in this regard).
- A polyinstantiated /tmp should be an option that is easy to configure for a novice sys-admin. There have been too many attacks on data confidentiality and system integrity based on file name race conditions in /tmp, this needs to be fixed and must be fixable by novice sys-admins.
- The capability system needs to be extended. 31 capabilities is not enough and the significant number of operations that are permitted by CAP_SYS_ADMIN leads to granting excessive privilege to programs.
- The use of Xen on servers such that a domU is always used for applications should become common. Then if a compromise is suspected there will be better options for investigation.
- SE Linux needs to be used more, particularly the strict policy and MCS. Use of the strict policy often reveals security flaws in other programs.
I’ll blog about each of these in detail at some future time.
This weekend I was in Sydney for Ruxcon. Ruxcon is a computer security conference with a focus on penetration testing and related skills.
The presentation on Unusual Bugs by Ilya van Sprudel was particularly interesting. He spoke about a number of issues that could do with some improvement in Linux, I will file some bug reports shortly.
There was a chilli eating contest. I was one of six people to enter. I survived the first two rounds and got onto the middle-strength chilli before giving up. There were 100 tickets to the Google party for the ~200 person conference and everyone who entered a contest got a ticket. My aim in the contest was to eat more chilli than I enjoy eating but less than the amount required to make me sick, with a secondary goal of tasting at least the second level of chilli. I achieved my goals and left the contest after tasting the second chilli.
One man appeared to be impressed by my chilli eating and was telling everyone that I am famous for eating chilli. It’s good to be famous for something in the computer security community. :-#
At the end of the conference there was a panel discussion that I was invited to attend. I had to leave early to catch my flight, at the time I left everyone who was on the panel had each finished a few drinks and a couple of new guys had just joined. I think I missed the most exciting part of the panel discussion.
Thanks to whoever paid for the drinks for panel members. Things were a little hectic when we were given the drinks and I forgot to thank whoever paid for them.
In other news Sydney trains are slow and unreasonably expensive, $13 to get from the airport to the SLUG meeting at St. Leonards seems excessive. With all the problems with Sydney roads they really need to get a better public transport system!
While in Sydney I attended a SLUG meeting and gave a short talk about Postal (my mail server benchmark suite). I will present a paper about Postal at the OSDC conference later this year.