BIND Configuration Files

I’ve recently been setting up more monitoring etc to increase the reliability of servers I run. One ongoing issue with computer reliability is any case where a person enters the same data in multiple locations, often people make mistakes and enter slightly different data which can give bad results.

For DNS you need to have at least 2 authoritative servers for each zone. I’ve written the below Makefile to extract the zone names from the primary server and generate a config file suitable for use on a secondary server. The next step is to automate this further by having the Makefile copy the config file to secondary servers and run “rndc reload”. Note that in a typical Debian configuration any user in group “bind” can write to BIND config files and reload the server configuration so this can be done without granting the script on the primary server root access on the secondary servers.

My blog replaces the TAB character with 8 spaces, you need to fix this up if you want to run the Makefile on your own system and also replace 10.10.10.10 with the IP address of your primary server.

all: other/secondary.conf

other/secondary.conf: named.conf.local Makefile
        for n in $$(grep ^zone named.conf.local | cut -f2 -d\"|sort) ; do echo "zone \"$$n\" {\n  type slave;\n  file \"$$n\";\n  masters { 10.10.10.10; };\n};\n" ; done > other/secondary.conf

Systemd Notes

A few months ago I gave a lecture about systemd for the Linux Users of Victoria. Here are some of my notes reformatted as a blog post:

Scripts in /etc/init.d can still be used, they work the same way as they do under sysvinit for the user. You type the same commands to start and stop daemons.

To get a result similar to changing runlevel use the “systemctl isolate” command. Runlevels were never really supported in Debian (unlike Red Hat where they were used for starting and stopping the X server) so for Debian users there’s no change here.

The command systemctl with no params shows a list of loaded services and highlights failed units.

The command “journalctl -u UNIT-PATTERN” shows journal entries for the unit(s) in question. The pattern uses wildcards not regexs.

The systemd journal includes the stdout and stderr of all daemons. This solves the problem of daemons that don’t log all errors to syslog and leave the sysadmin wondering why they don’t work.

The command “systemctl status UNIT” gives the status and last log entries for the unit in question.

A program can use ioctl(fd, TIOCSTI, …) to push characters into a tty buffer. If the sysadmin runs an untrusted program with the same controlling tty then it can cause the sysadmin shell to run hostile commands. The system call setsid() to create a new terminal session is one solution but managing which daemons can be started with it is difficult. The way that systemd manages start/stop of all daemons solves this. I am glad to be rid of the run_init program we used to use on SE Linux systems to deal with this.

Systemd has a mechanism to ask for passwords for SSL keys and encrypted filesystems etc. There have been problems with that in the past but I think they are all fixed now. While there is some difficulty during development the end result of having one consistent way of managing this will be better than having multiple daemons doing it in different ways.

The commands “systemctl enable” and “systemctl disable” enable/disable daemon start at boot which is easier than the SysVinit alternative of update-rc.d in Debian.

Systemd has built in seat management, which is not more complex than consolekit which it replaces. Consolekit was installed automatically without controversy so I don’t think there should be controversy about systemd replacing consolekit.

Systemd improves performance by parallel start and autofs style fsck.

The command systemd-cgtop shows resource use for cgroups it creates.

The command “systemd-analyze blame” shows what delayed the boot process and
systemd-analyze critical-chain” shows the critical path in boot delays.

Sysremd also has security features such as service private /tmp and restricting service access to directory trees.

Conclusion

For basic use things just work, you don’t need to learn anything new to use systemd.

It provides significant benefits for boot speed and potentially security.

It doesn’t seem more complex than other alternative solutions to the same problems.

https://wiki.debian.org/systemd

http://freedesktop.org/wiki/Software/systemd/Optimizations/

http://0pointer.de/blog/projects/security.html

Inteltech/Clicksend SMS Script

USER=username
API_KEY=1234ABC
OUTPUTDIR=/var/spool/sms
LOG_SERVICE=local1

I’ve just written the below script to send SMS via the inteltech.com/clicksend.com service. It takes the above configuration in /etc/sms-pass.cfg where the username is assigned with the clicksend web page and the API key is a long hex string that clicksend provides as a password. The LOG_SERVICE is which syslog service to use for the log messages, on systems that are expected to send many messages I use the service “local1” and I use “user” for development systems.

I hope this is useful to someone, and if you have any ideas for improvement then please let me know.

#!/bin/sh
# $1 is destination number
# text is on standard input
# standard output gives message ID on success, and 0 is returned
# standard error gives error from server on failure, and 1 is returned

. /etc/sms-pass.cfg
OUTPUT=$OUTPUTDIR/out.$$
TEXT=`tr "[:space:]" + | cut -c 1-159`

logger -t sms -p $LOG_SERVICE.info "sending message to $1"
wget -O $OUTPUT "https://api.clicksend.com/http/v2/send.php?method=http&username=$USER&key=$API_KEY&to=$1&message=$TEXT" > /dev/null 2> /dev/null

if [ "$?" != "0" ]; then
  echo "Error running wget" >&2
  logger -t sms -p $LOG_SERVICE.err "failed to send message \"$TEXT\" to $1 – wget error"
  exit 1
fi

if ! grep -q ^.errortext.Success $OUTPUT ; then
  cat $OUTPUT >&2
  echo >&2
  ERR=$(grep ^.errortext $OUTPUT | sed -e s/^.errortext.// -e s/..errortext.$//)
  logger -t sms -p $LOG_SERVICE.err "failed to send message \"$TEXT\" to $1 – $ERR"
  rm $OUTPUT
  exit 1
fi

ID=$(grep ^.messageid $OUTPUT | sed -e s/^.messageid.// -e s/..messageid.$//)
rm $OUTPUT

logger -t sms -p $LOG_SERVICE.info "sent message to $1 with ID $ID"
exit 0

Modern Swap Use

A while ago I wrote a blog post debunking the idea that swap should be twice as large as RAM [1]. The issue of swap size has come up for discussion in a mailing list so it seems that it’s worth another blog post.

Swap Size

In my previous post I suggested making swap space equal to RAM size for systems with less than 1G of RAM and half RAM size for systems with 2-4G of RAM with a maximum of 2G for any system. Generally it’s better to have the kernel Oom handler kill a memory hungry process than to have the entire system go so slow that a hardware reset is needed. I wrote memlockd to lock the important system programs and the shared objects and data files they use into RAM to make it easier to recover from some bad paging situations [2], but even that won’t always make it possible to login to a thrashing system in a reasonable amount of time.

But it’s not always good to reduce swap size. Sometimes if you don’t have enough swap then performance sucks in situations where it would be OK if there was more swap. One factor in this regard is that pages mapped read-only from files are discarded when there is great memory pressure and no swap space to page-out the read-write pages. This can mean that you get thrashing among memory for executables and shared objects while there’s lots of unused data pages in RAM that can’t be paged out. One noteworthy corner case in this regard is Chromium which seems to take a lot of RAM if you have many tabs open for a long time.

Another factor is that there is a reasonable amount of memory allocated which will almost never get used (EG all the X based workstations which have 6 gettys running on virtual consoles which will almost never be used). So a system which has a large amount of RAM relative to it’s use (EG a single-user workstation with 8G of RAM) can still benefit from having a small amount of swap to allow larger caches. One workstation I run has 8G of RAM for a single user and typically has about 200M of the 1G swap space in use. It doesn’t have enough swap to make much difference if really memory hungry programs are run, but having 4G of memory for cache instead of 3.8G might make a difference to system performance. So even systems which can run without using any swap can still be expected to give better performance if there is some swap space for programs that are very inactive.

I have some systems with as much 4G of swap, but they are for corner cases where swap is allocated but there isn’t a lot of paging going on. For example I have a workstation with 3G of RAM and 4G of swap for the benefit of Chromium.

Hardware Developments

Disk IO performance hasn’t increased much when compared to increases in RAM size over the last ~15 years. A low-end 1988 era hard drive could sustain 512KB/s contiguous transfer rates and had random access times of about 28ms. A modern SATA disk will have contiguous transfer rates getting close to 200MB/s and random access times less than 4ms. So we are looking at about 400* the contiguous performance and about 10* the random access since 1988. In 1988 2MB was a lot of RAM for a PC, now no-one would buy a new PC with less than 4G – so we are looking at something like 2000* the RAM size in the same period.

When comparing with 1993 (when I first ran a Linux server 24*7) it was 4M of RAM, maybe 15ms random access, and maybe 1MB/s contiguous IO – approximately double everything I had in 1988. But I’ll use the numbers from 1988 because I’m more certain of them even though I never ran an OS capable of proper memory management on the 1988 PC.

In the most unrealistically ideal situation paging IO will be a factor of 5* more expensive now relatively than it was in 1988 (RAM size increase by a factor of 2000 divided by contiguous IO performance increase by a factor of 400). But in a more realistic situation (random IO) it will be 200* more expensive relatively than it was. In the early 90s a Linux system with swap use equal or greater than RAM could perform well (my recollection is that a system with 4M of RAM and 4M of active swap performed well and the same system with 8M of active swap was usable). But nowadays there’s no chance of a system that has 4G of RAM and 4G of swap in active use being usable in any way unless you have some corner case of an application allocating RAM and not using it much.

Note that 4G of RAM doesn’t make a big system by today’s standards, so it might be reasonable to compare 2M in 1988 to 8G or 16G today. But I think that assuming 4G of RAM for the purpose of comparison makes my point.

Using Lots of Swap

It is probably possible to have some background Chromium tabs using 4G of paged out RAM with good system performance. I’ve had a system use 2G of swap for Chromium and perform well overall. There are surely many other ways that a user can launch processes that use a lot of memory and not use them actively. But this will probably require that the user know how the system works and alter their usage patterns. If you have a lot of swap in use and one application starts to run slowly you really don’t want to flip to another application which would only make things worse.

If you use tmpfs for /tmp then you need enough swap to store everything that is written to /tmp. There is no real upper limit to how much data that could be apart from the fact that applications and users generally don’t want to write big data files that disappear on reboot. I generally only use a tmpfs for /tmp on servers. Programs that run on servers tend not to store a large amount of data in /tmp and that data is usually very temporary, unlike workstation users who store large video files in /tmp and leave them there until the next reboot.

Are there any other common corner cases that allow using a lot of swap space without serious performance problems?

My Swap Space Allocation

When I setup Linux workstations for other people I generally use 1G of swap. For a system that often has multiple users (via the user-switch feature) I use 2G of swap if it only has 3G of RAM (I run many systems which are limited to 3G of RAM by their chipset).

For systems that I personally use I generally allocate 4G of swap, this is to cater to excessive RAM use by Chromium and also because I test out random memory hungry programs and can’t easily predict how much swap space will be needed – a typical end-user desktop system is a lot more predictable than a workstation used for software development.

For servers I generally allocate 512M of swap space. That’s enough to page out things that aren’t used and make space for cache memory. If that turns out to be inadequate then it’s probably best to just let the watchdog daemon reboot the system.

Using BTRFS

I’ve just installed BTRFS on some systems that matter to me. It is still regarded as experimental but Oracle supports it with their kernel so it can’t be too bad – and it’s almost guaranteed that anything other than BTRFS or ZFS will lose data if you run as many systems as I do. Also I run lots of systems that don’t have enough RAM for ZFS (4G isn’t enough for ZFS in my tests). So I have to use BTRFS.

BTRFS and Virtual Machines

I’m running BTRFS for the DomUs on a virtual server which has 4G of RAM (and thus can’t run ZFS). The way I have done this is to use ext4 on Linux software RAID-1 for the root filesystem on the Dom0 and use BTRFS for the rest. For BTRFS and virtual machines there seem to be two good options given that I want BTRFS to use it’s own RAID-1 so that it can correct errors from a corrupted disk. One is to use a single BTRFS filesystem with RAID-1 for all the storage and then have each VM use a file on that big BTRFS filesystem for all it’s storage. The other option is to have each virtual machine run BTRFS RAID-1.

I’ve created two LVM Volume Groups (VGs) named diska and diskb, each DomU has a Logical Volume (LV) from each VG and runs BTRFS. So if a disk becomes corrupt the DomU will have to figure out what the problem is and fix it.

#!/bin/bash
for n in $(xm list|cut -f1 -d\ |egrep -v ^Name\|^Domain-0) ; do
  echo $n
  ssh $n "btrfs scrub start -B -d /"
done

I use the above script in a cron job from the Dom0 to scrub the BTRFS filesystems in the DomUs. I use the -B option so that I will receive email about any errors and so that there won’t be multiple DomUs scrubbing at the same time (which would be really bad for performance).

BTRFS and Workstations

The first workstation installs of BTRFS that I did were similar to installations of Ext3/4 in that I had multiple filesystems on LVM block devices. This caused all the usual problems of filesystem sizes and also significantly hurt performance (sync seems to perform very badly on a BTRFS filesystem and it gets really bad with lots of BTRFS filesystems). BTRFS allows using subvolumes for snapshots and it’s designed to handle large filesystems so there’s no reason to have more than one filesystem IMHO.

It seems to me that the only benefit in using multiple BTRFS filesystems on a system is if you want to use different RAID options. I presume that eventually the BTRFS developers will support different RAID options on a per-subvolume basis (they seem to want to copy all ZFS features). I would like to be able to configure /home to use 3 copies of all data and metadata on a workstation that only has a single disk.

Currently I have some workstations using BTRFS with a BTRFS RAID-1 configuration for /home and a regular non-RAID configuration for everything else. But now it seems that this is a bad idea, I would be better off just using a single copy of all data on workstations (as I did for everything on workstations for the previous 15 years of running Linux desktop systems) and make backups that are frequent enough to not have a great risk.

BTRFS and Servers

One server that I run is primarily used as an NFS server and as a workstation. I have a pair of 3TB SATA disks in a BTRFS RAID-1 configuration mounted as /big and with subvolumes under /big for the various NFS exports. The system also has a 120G Intel SSD for /boot (Ext4) and the root filesystem which is BTRFS and also includes /home. The SSD gives really good read performance which is largely independent of what is done with the disks so booting and workstation use are very fast even when cron jobs are hitting the file server hard.

The system has used a RAID-1 array of 1TB SATA disks for all it’s storage ever since 1TB disks were big. So moving to a single storage device for /home is a decrease in theoretical reliability (in addition to the fact that a SSD might be less reliable than a traditional disk). The next thing that I am going to do is to install cron jobs that backup the root filesystem to something under /big. The server in question isn’t used for anything that requires high uptime, so if the SSD dies entirely and I need to replace it with another boot device then it will be really annoying but it won’t be a great problem.

Snapshot Backups

One of the most important uses of backups is to recover from basic user mistakes such as deleting the wrong file. To deal with this I wrote some scripts to create backups from a cron job. I put the snapshots of a subvolume under a subvolume named “backup“. A common use is to have everything on the root filesystem, /home as a subvolume, /home/backup as another subvolume, and then subvolumes for backups such as /home/backup/2012-12-17, /home/backup/2012-12-17:00:15, and /home/backup/2012-12-17:00:30. I make /home/backup world readable so every user can access their own backups without involving me, of course this means that if they make a mistake related to security then I would have to help them correct it – but I don’t expect my users to deal with security issues, if they accidentally grant inappropriate access to their files then I will be the one to notice and correct it.

Here is a script I name btrfs-make-snapshot which has an optional first parameter “-d” to cause it do just display the btrfs commands it would run and not actually do anything. The second parameter is either “minutes” or “days” depending on whether you want to create a snapshot on a short interval (I use 15 minutes) or a daily snapshot. All other parameters are paths for subvolumes that are to be backed up:

#!/bin/bash
set -e

# usage:
# btrfs-make-snapshot [-d] minutes|days paths
# example:
# btrfs-make-snapshot minutes /home /mail

if [ "$1" == "-d" ]; then
  BTRFS="echo btrfs"
  shift
else
  BTRFS=/sbin/btrfs
fi

if [ "$1" == "minutes" ]; then
  DATE=$(date +%Y-%m-%d:%H:%M)
else
  DATE=$(date +%Y-%m-%d)
fi
shift

for n in $* ; do
  $BTRFS subvol snapshot -r $n $n/backup/$DATE
done

Here is a script I name btrfs-remove-snapshots which removes old snapshots to free space. It has an optional first parameter “-d” to cause it do just display the btrfs commands it would run and not actually do anything. The next parameters are the number of minute based and day based snapshots to keep (I am currently experimenting with 100 100 for /home to keep 15 minute snapshots for 25 hours and daily snapshots for 100 days). After that is a list of filesystems to remove snapshots from. The removal will be from under the backup subvolume of the path in question.

#!/bin/bash
set -e

# usage:
# btrfs-remove-snapshots [-d] MINSNAPS DAYSNAPS paths
# example:
# btrfs-remove-snapshots 100 100 /home /mail

if [ "$1" == "-d" ]; then
  BTRFS="echo btrfs"
  shift
else
  BTRFS=/sbin/btrfs
fi

MINSNAPS=$1
shift
DAYSNAPS=$1
shift

for DIR in $* ; do
  BASE=$(echo $DIR | cut -c 2-200)
  for n in $(btrfs subvol list $DIR|grep $BASE/backup/.*:|head -n -$MINSNAPS|sed -e "s/^.* //"); do
    $BTRFS subvol delete /$n
  done
  for n in $(btrfs subvol list $DIR|grep $BASE/backup/|grep -v :|head -n -$MINSNAPS|sed -e "s/^.* //"); do
    $BTRFS subvol delete /$n
  done
done

A Warning

The Debian/Wheezy kernel (based on the upstream kernel 3.2.32) doesn’t seem to cope well when you run out of space by making snapshots. I have a filesystem that I am still trying to recover after doing that.

I’ve just been buying larger storage devices for systems while migrating them to BTRFS, so I should be able to avoid running out of disk space again until I can upgrade to a kernel that fixes such bugs.

Parsing Daemontools/Multilog dates in Shell Script

I run some servers that use the DJB Daemontools to manage their daemons [1]. This is something I would have changed years ago if given a chance because DJB software always seems to do things in a different way to other Unix software and causes pain. Unfortunately when you have a lot of semi-embedded systems that have intermittent net access it’s really not easy to change things, and having staff who aren’t computer experts who have been trained to use certain DJB software makes it even more difficult.

Daemontools uses multilog [2] to manage it’s logging, this gives dates of the format @400000004deedcea1e4a18d4. While DJB has written a tool to parse this it’s not always convenient, and I don’t want to install DJB software on every system that might be used for reading logs (among other things DJB software is not included in any popular distribution).

George Bernard Shaw says that “All progress depends on the unreasonable man” [3], of course he never participated in a large-scale software development project. In the modern age progress usually depends on people who can work with others, which is why DJB software doesn’t get used much – for every DJB program there is a similar program written by someone else that works about equally well on it’s own and is more than 10* more popular because of better interoperability.

So I wrote the following script to convert DJB dates to regular dates. It takes a DJB format date as the first command-line parameter as I generally just paste the relevant date into a different window. At some future time I may write a program to parse an entire log file and convert all the dates but I haven’t had a need for it yet. I think that I’ve done the hardest work involved in writing such a parser so someone else can use this as a starting point if they have such a need.

#!/bin/bash
set -e
DATE=$(echo $1|cut -c 10-17)
SECS=$(echo -e ibase=16\\n$(echo $DATE|tr "[a-z]" "[A-Z]")|bc)
exec date -d "1970-01-01 $SECS sec utc"

Maintaining Screen Output

In my post about getting started with KVM I noted the fact that I had problems keeping screen output after the program exits [1].

The following snippet of shell code demonstrates the solution I’ve discovered for this problem. It determines whether SCREEN is the parent process of the shell script and if so it sleeps for 60 seconds before exiting so I can see the KVM error messages. The other option is for the script to call “exec bash” to give me a new shell in the same window. Note that if I start a screen session and then run my KVM script I don’t want it to do anything special on exit as I will return to the command-line in the same window. If I run “exec kvm-unstable” or have a system boot script run “start-stop-daemon -S -c USER --exec /usr/bin/screen -- -S kvm-unstable -d -m /usr/local/bin/kvm-unstable” then on exit I will be able to see what happened.

#!/bin/bash
set -e
kvm ETC || echo “KVM gave an error return code”
COUNT=$(ps aux|grep $PPID|grep SCREEN|wc -l)
if [ "$COUNT" = "1" ]; then
  echo "screen is the parent"
  sleep 60
else
  echo no screen
fi

Update: Thanks to John for the Slee for suggesting the following:
#!/bin/bash
set -e
kvm ETC || echo “KVM gave an error return code”
if grep -q SCREEN /proc/$PPID/cmdline ; then
  echo "screen is the parent"
  sleep 60
else
  echo no screen
fi

SCO Unix and Proftpd

I recently had the misfortune to be compelled to install Proftpd on SCO Unix. There’s nothing wrong with Proftpd of course, but everything is wrong with SCO.

LDFLAGS=-L/usr/ucblib ./configure –libdir=/usr/ucblib --enable-builtin-getnameinfo

To get it to compile the above ./configure line is needed. It uses /usr/ucblib because having the basic BSD sockets API residing in a non-default directory makes things more exciting. The --enable-builtin-getnameinfo option is needed because the getnameinfo library call in SCO Unix doesn’t work properly. On a real OS I would have downloaded the source to libc to figure out where the problem might be, but on SCO I had no option other than to disable that functionality.

Then once I got it working I had to put UseIPv6 off in the configuration file, otherwise I got the error No Route to Host (from memory) when I tried to start proftpd.

Finally the process was dying from an unhandled SIGALRM every 10 seconds. According to truss (a vastly inferior equivalent to strace) the sigaction() system call was being entered four times to handle SIGALRM (as was reported for every time the application code called sigaction() but the default action of terminating the application remained in place. I ended up running proftpd from inetd which removed the need for the alarm handling code. Then it all worked.

SCO sucks.

Last month Darl McBride was terminated from his position with the SCO group [1]. It’s unfortunate that Darl wasn’t terminated a lot earlier, if so then maybe SCO could have focussed on improving the quality of their products and delivering value for stock holders. Hopefully Darl will be sued by some of the people who lost money when SCO went bankrupt.

It would be good if Darl could also be sued by companies that lost money due to the poor quality of SCO Unix – that means every company that runs SCO servers or that writes software for it. The truly awful nature of SCO Unix is in large part due to the fact that all of SCO’s money was spent on litigation that was well known to be baseless.

Some Tips for Shell Code that Won’t Destroy Your OS

When writing a shell script you need to take some care to ensure that it won’t run amok. Extra care is needed for shell scripts that run as root, firstly because of the obvious potential for random destruction, and secondly because of the potential for interaction between accounts that can cause problems.

  • One possible first step towards avoiding random destruction is to start your script with “#/bin/sh -e” instead of “#/bin/sh“, this means that the script will exit on an unexpected error, which is generally better than continuing merrily along to destroy vast swathes of data. Of course sometimes you will expect an error, in which case you can use “/usr/local/bin/command-might-fail || true” to make it not abort on a command that might fail.
  • #!/bin/sh -e
    cd /tmp/whatever
    rm -rf *
    #!/bin/sh
    cd /tmp/whatever || exit 1
    rm -rf *

    Instead of using the “-e” switch to the shell you can put “|| exit 1” after a command that really should succeed. For example neither of the above scripts is likely to destroy your system, while the following script is very likely to destroy your system:
    #!/bin/sh
    cd /tmp/whatever
    rm -rf *
  • Also consider using absolute paths. “rm -rf /tmp/whatever/*” is as safe as the above option but also easier to read – avoiding confusion tends to improve the reliability of the system. Relative paths are most useful for humans doing typing, when a program is running there is no real down-side to using long absolute paths.
  • Shell scripts that cross account boundaries are a potential cause of problems, for example if a script does “cd /home/user1” instead of “cd ~user1” then if someone in the sysadmin team moves the user’s home directory to /home2/user1 (which is not uncommon when disk space runs low) then things can happen that you don’t expect – and we really don’t want unexpected things happening as root! Most shells don’t support “cd ~$1“, but that doesn’t force you to use “cd /home/$1“, instead you can use some shell code such as the following:
    #!/bin/sh
    HOME=`grep ^$1 /etc/passwd|head -1|cut -f6 -d:`
    if [ "$HOME" = "" ]; then
      echo "no home for $1"
      exit 1
    fi
    cd ~


    I expect that someone can suggest a better way of doing that. My point is not to try and show the best way of solving the problem, merely to show that hard coding assumptions about paths is not necessary. You don’t need to solve a problem in the ideal way, any way that doesn’t have a significant probability of making a server unavailable and denying many people the ability to do their jobs will do. Also consider using different tools, zsh supports commands such as “cd ~$1“.
  • When using a command such as find make sure that you appropriately limit the results, in the case of find that means using options such as -xdev, -type, and -maxdepth. If you mistakenly believe that permission mode 666 is appropriate for all files in a directory then it won’t do THAT much harm. But if your find command goes wrong and starts applying such permissions to directories and crosses filesystem boundaries then your users are going to be very unhappy.
  • Finally when multiple scripts use the same data consider using a configuration file. If you feel compelled to do something grossly ugly such as writing a dozen expect scripts which use the root password then at least make it an entry in a configuration file so that it can be changed in one place. It seems that every time I get a job working on some systems that other people have maintained there is at least one database, LDAP directory, or Unix root account for which the password can’t be changed because no-one knows how many scripts have it hard-coded. It’s usually the most important server, database, or directory too.

Please note that nothing in this post is theoretical, it’s all from real observations of real systems that have been broken.

Also note that this is not an attempt at making an exhaustive list of ways that people may write horrible scripts, merely enough to demonstrate the general problem and encourage people to think about ways to solve the general problems. But please submit your best examples of how scripts have broken systems as comments.

Time Zones and Remote Servers

It’s widely regarded that the best practice is to set the time zone of a server to UTC if people are going to be doing sys-admin work from various countries. I’m currently running some RHEL4 servers that are set to Los Angeles time. So I have to convert the time from Melbourne time to UTC and then from UTC to LA time when tracking down log entries. This isn’t really difficult for recent times (within the last few minutes) as my KDE clock applet allows me to select various time zones to display on a pop-up. For other times I can use the GNU date command to convert from other time zones to the local zone of the machine, for example the command date -d "2008-08-06 10:57 +1000" will display the current Melbourne time (which is in the +1000 time zone) converted to the local time zone. But it is still painful.

In RHEL5, CentOS 5, and apparently all versions of Fedora newer Fedora Core 4 (including Fedora Core 4 updates) the command system-config-date allows you to select Etc/GMT as the time zone to get GMT. For reference selecting London is not a good option, particularly at the moment as it’s apparently daylight savings time there.

For RHEL4 and CentOS 4 the solution is to edit /etc/sysconfig/clock and change the first line to ZONE="Etc/GMT" (the quotes are important), and then run the command ln -sf /usr/share/zoneinfo/Etc/GMT /etc/localtime. Thanks to the Red Hat support guy who found the solution to this, it took a while but it worked in the end! Hopefully this blog post will allow others to fix this without needing to call Red Hat.

In Debian the command tzconfig allows you to select 12 (none of the above) and then GMT or UTC to set the zone. This works in Etch, I’m not sure about earlier versions (tzconfig always worked but I never tried setting UTC). In Lenny the tzconfig command seems to have disappeared, now to configure the time zone you use the command dpkg-reconfigure tzdata which has an option og Etc at the end of the list.

Updated to describe how to do this in Lenny, thanks for the comments.