LUV Server Upgrade to Jessie

On Sunday night I started the process of upgrading the LUV server to Debian/Jessie from Debian/Wheezy. My initial plan was to just upgrade Apache first but dependencies required upgrading systemd too.

One problem I’ve encountered in the past is that the Wheezy version of systemd will often hang on an upgrade to a newer version. Generally the solution to this is to run “systemctl daemon-reexec” from another terminal. The problem in this case was that not all the libraries needed for systemd had been installed, so systemd could re-exec itself but immediately aborted. The kernel really doesn’t like it when process 1 aborts repeatedly and apparently immediately hanging is the result. At the time I didn’t know this, all I knew was that my session died and the server stopped responding to pings immediately after I requested a reexec.

The LUV server is hosted at VPAC for free. As their staff have actual work to do they couldn’t spend a lot of time working on the LUV server. They told me that the screen was flickering and suspected a VGA cable. I got to the VPAC server room with the spare LUV server (LUV had been given 3 almost identical Sun servers from Barwon Water) at 16:30. By 17:30 I had fixed the core problem (boot with “init=/bin/bash“, mount the root filesystem rw, finish the upgrade of systemd and it’s dependencies, and then reboot normally). That got it into a stage where the Xen server for Wikimedia Au was working but most LUV functionality wasn’t working.

By 23:00 on Monday I had the full list server functionality working for users, this is the main feature that users want when it’s not near a meeting time. I can’t remember whether it was Monday night or Tuesday morning when I got the Drupal site going (the main LUV web site). Last night at midnight I got the last of the Mailman administrative interface going, I admit I could have got it going a bit earlier by putting SE Linux in permissive mode, but I don’t think that the members would have benefited from that (I’ll upload a SE Linux policy package that gets Mailman working on Jessie soon).

Now it’s Wednesday and I’m still fixing some cron jobs. Along the way I noticed some problems with excessive disk space use that I’m fixing now and I’ve also removed some Wikimedia related configuration files that were obsolete and would have prevented anyone from using a wikimedia.org.au address to subscribe to the LUV mailing lists.

Now I believe that everything is working correctly and generally working better than before.

Lessons Learned

While Sunday night wasn’t a bad time to start the upgrade it wasn’t the best. If I had started the upgrade on Monday morning there would have been less down-time. Another possibility might be to do the upgrade while near the VPAC office during business hours, I could have started the upgrade while at a nearby cafe and then visited the server room immediately if something went wrong.

Doing an upgrade on a day when there’s no meeting within a week was a good choice. It wasn’t really a conscious choice as I’m usually doing other LUV work near the meeting day which precludes doing other LUV work that doesn’t need to be done soon. But in future it would be best to consciously plan upgrades for a date when users aren’t going to need the service much.

While the Wheezy systemd bug is unlikely to ever be fixed there are work-arounds that shouldn’t result in a broken server. At the moment it seems that the best option would be to kill -9 the systemctl processes that hang until the packages that systemd depends on are installed. The problem is that the upgrade hangs while the new systemctl tries to tell the old systemd to restart daemons. If we can get past that to the stage where the shared objects are installed then it should be ok.

The Apache upgrade from 2.2.x to 2.4.x changed the operation of some access control directives and it took me some time to work out how to fix that. Doing a Google search on the differences between those would have led me to the Apache document about upgrading from 2.2 to 2.4 [1]. That wouldn’t have prevented some down-time of the web sites but would have allowed me to prepare for it and to more quickly fix the problems when they became apparent. Also the rather confusing configuration of the LUV server (supporting many web sites that are no longer used) didn’t help things. I think that removing cruft from an installation before an upgrade would be better than waiting until after things break.

Next time I do an upgrade of such a server I’ll write notes about it while I go. That will give a better blog post about it if it becomes newsworthy enough to be blogged about and also more opportunities to learn better ways of doing it.

Sorry for the inconvenience.

Mail Server Training

Today I ran a hands-on training session on configuring a MTA with Postfix and Dovecot for LUV. I gave each student a virtual machine running Debian/Jessie with full Internet access and instructions on how to configure it as a basic mail server. Here is a slightly modified set of instructions that anyone can do on their own system.

Today I learned that documentation that includes passwords on a command-line should have quotes around the password, one student used a semi-colon character in his password which caused some confusion (it’s the command separator character in BASH). I also discovered that trying to just tell users which virtual server to login to is prone to errors, in future I’ll print out a list of user-names and passwords for virtual servers and tear off one for each student so there’s no possibility of 2 users logging in to the same system.

I gave each student a sub-domain of unixapropos.com (a zone that I use for various random sysadmin type things). I have changed the instructions to use example.com which is the official address for testing things (or you could use any zone that you use). The test VMs that I setup had a user named “auser”, the documentation assumes this account name. You could change “auser” to something else if you wish.

Below are all the instructions for anyone who wants to try it at home or setup virtual machines and run their own training session.

Basic MTA Configuration

  1. Run “apt-get install postfix” to install Postfix, select “Internet Site” for the type of mail configuration and enter the domain name you selected for the mail name.
  2. The main Postfix configuration file is /etc/postfix/main.cf. Change the myhostname setting to the fully qualified name of the system, something like mta.example.com.
    You can edit /etc/postfix/main.cf with vi (or any other editor) or use the postconf command to change it, eg “postconf -e myhostname=mta.example.com“.
  3. Add “home_mailbox=Maildir/” to the Postfix configuration to make it deliver to a Maildir spool in the user’s home directory.
  4. Restart Postfix to apply the changes.
  5. Run “apt-get install swaks libnet-ssleay-perl” to install swaks (a SMTP test tool).
  6. Test delivery by running the command “swaks -f auser@example.com -t auser@example.com -s localhost“. Note that swaks displays the SMTP data so you can see exactly what happens and if something goes wrong you will see everything about the error.
  7. Inspect /var/log/mail.log to see the messages about the delivery. View the message which is in ~auser/Maildir/new.
  8. When other students get to this stage run the same swaks command but with the -t changed to the address in their domain, check the mail.log to see that the messages were transferred and view the mail with less to see the received lines. If you do this on your own specify a recipient address that’s a regular email address of yours (EG a Gmail account).

Basic Pop/IMAP Configuration

  1. Run “apt-get install dovecot-pop3d dovecot-imapd” to install Dovecot POP and IMAP servers.
    Run “netstat -tln” to see the ports that have daemons listening on them, observe that ports 110 and 143 are in use.
  2. Edit /etc/dovecot/conf.d/10-mail.conf and change mail_location to “maildir:~/Maildir“. Then restart Dovecot.
  3. Run the command “nc localhost 110” to connect to POP, then run the following commands to get capabilities, login, and retrieve mail:
    user auser
    pass WHATEVERYOUMADEIT
    capa
    list
    retr 1
    quit
  4. Run the command “nc localhost 143” to connect to IMAP, then run the following commands to list capabilities, login, and logout:
    a capability
    b login auser WHATEVERYOUMADEIT
    c logout
  5. For the above commands make note of the capabilities, we will refer to that later.

Now you have a basically functional mail server on the Internet!

POP/IMAP Over SSL

To avoid password sniffing we need to use SSL. To do it properly requires obtaining a signed key for a DNS address but we can do the technical work with the “snakeoil” certificate that is generated by Debian.

  1. Edit /etc/dovecot/conf.d/10-ssl.conf and change “ssl = no” to “ssl = required“. Then add the following 2 lines:
    ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
    ssl_key = </etc/ssl/private/ssl-cert-snakeoil.key
    1. Run “netstat -tln” and note that ports 993 and 995 are not in use.
    2. Edit /etc/dovecot/conf.d/10-master.conf and uncomment the following lines:
      port = 993
      ssl = yes
      port = 995
      ssl = yes
    3. Restart Dovecot, run “netstat -tln” and note that ports 993 and 995 are in use.
  2. Run “nc localhost 110” and “nc localhost 143” as before, note that the capabilities have changed to include STLS/STARTTLS respectively.
  3. Run “gnutls-cli --tofu 127.0.0.1 -p 993” to connect to the server via IMAPS and “gnutls-cli --tofu 127.0.0.1 -p 995” to connect via POP3S. The --tofu option means to “Trust On First Use”, it stores the public key in ~/.gnutls and checks it the next time you connect. This allows you to safely use a “snakeoil” certificate if all apps can securely get a copy of the key.

Postfix SSL

  1. Edit /etc/postfix/main.cf and add the following 4 lines:
    smtpd_tls_received_header = yes
    smtpd_tls_loglevel = 1
    smtp_tls_loglevel = 1
    smtp_tls_security_level = may

    Then restart Postfix. This makes Postfix log TLS summary messages to syslog and in the Received header. It also permits Postfix to send with TLS.
  2. Run “nc localhost 25” to connect to your SMTP port and then enter the following commands:
    ehlo test
    quit

    Note that the response to the EHLO command includes 250-STARTTLS, this is because Postfix was configured with the Snakeoil certificate by default.
  3. Run “gnutls-cli --tofu 127.0.0.1 -p 25 -s” and enter the following commands:
    ehlo test
    starttls
    ^D

    After the CTRL-D gnutls-cli will establish a SSL connection.
  4. Run “swaks -tls -f auser@example.com -t auser@example.com -s localhost” to send a message with SSL encryption. Note that swaks doesn’t verify the key.
  5. Try using swaks to send messages to other servers with SSL encryption. Gmail is one example of a mail server that supports SSL which can be used, run “swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com” to send TLS (encapsulated SSL) mail to Gmail via swaks. Also run “swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s localhost” to send via your new mail server (which should log that it was a TLS connection from swaks and a TLS connection to Gmail).

SASL

SASL is the system of SMTP authentication for mail relaying. It is needed to permit devices without fixed IP addresses to send mail through a server. The easiest way of configuring Postfix SASL is to have Dovecot provide it’s authentication data to Postfix. Among other things if you change Dovecot to authenticate in another way you won’t need to make any matching changes to Postfix.

  1. Run “mkdir -p /var/spool/postfix/var/spool” and “ln -s ../.. /var/spool/postfix/var/spool/postfix“, this allows parts of Postfix to work with the same configuration regardless of whether they are running in a chroot.
  2. Add the following to /etc/postfix/main.cf and restart Postfix:
    smtpd_sasl_auth_enable = yes
    smtpd_sasl_type = dovecot
    smtpd_sasl_path = /var/spool/postfix/private/auth
    broken_sasl_auth_clients = yes
    smtpd_sasl_authenticated_header = yes
  3. Edit /etc/dovecot/conf.d/10-master.conf, uncomment the following lines, and then restart Dovecot:
    unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    }
  4. Edit /etc/postfix/master.cf, uncomment the line for the submission service, and restart Postfix. This makes Postfix listen on port 587 which is allowed through most firewalls.
  5. From another system (IE not the virtual machine you are working on) run “swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s YOURSERVER and note that the message is rejected with “Relay access denied“.
  6. Now run “swaks -tls --auth-user auser --auth-password WHATEVER -f auser@example.com -t YOURREALADDRESS -s YOURSERVER” and observe that the mail is delivered (subject to anti-spam measures at the recipient).
  7. Configuring a MUA

    If every part of the previous 3 sections is complete then you should be able to setup your favourite MUA. Use “auser” as the user-name for SMTP and IMAP, mail.example.com for the SMTP/IMAP server and it should just work! Of course you need to use the same DNS server for your MUA to have this just work. But another possibility for testing is to have the MUA talk to the server by IP address not by name.

Public Lectures About FOSS

Eventbrite

I’ve recently started using the Eventbrite Web site [1] and the associated Eventbrite Android app [2] to discover public events in my area. Both the web site and the Android app lack features for searching (I’d like to save alerts for my accounts and have my phone notify me when new events are added to their database) but it is basically functional. The main issue is content, Eventbrite has a lot of good events in their database (I’ve got tickets for 6 free events in the next month). I assume that Eventbrite also has many people attending their events, otherwise the events wouldn’t be promoted there.

At this time I haven’t compared Eventbrite to any similar services, Eventbrite events have taken up much of my available time for the next 6 weeks (I appreciate the button on the app to add an entry to my calendar) so I don’t have much incentive to find other web sites that list events. I would appreciate comments from users of competing event registration systems and may write a post in future comparing different systems. Also I have only checked for events in Melbourne, Australia as I don’t have any personal interest in events in other places. For the topic of this post Eventbrite is good enough, it meets all requirements for Melbourne and I’m sure that if it isn’t useful in other cities then there are competing services.

I think that we need to have free FOSS events announced through Eventbrite. We regularly have experts in various fields related to FOSS visiting Melbourne who give a talk for the Linux Users of Victoria (and sometimes other technical groups). This is a good thing but I think we could do better. Most people in Melbourne probably won’t attend a LUG meeting and if they did they probably wouldn’t find it a welcoming experience.

Also I recommend that anyone who is looking for educational things to do in Melbourne visit the Eventbrite web site and/or install the Android app.

Accessible Events

I recently attended an Eventbrite event where a professor described the work of his research team, it was a really good talk that made the topic of his research accessible to random members of the public like me. Then when it came to question time the questions were mostly opinion pieces disguised as questions which used a lot of industry specific jargon and probably lost the interest of most people in the audience who wasn’t from the university department that hosted the lecture. I spent the last 15 minutes in that lecture hall reading Wikipedia and resisted the temptation to load an Android game.

Based on this lecture (and many other lectures I’ve seen) I get the impression that when the speaker or the MC addresses a member of the audience by name (EG “John Smith has a question”) then it’s strongly correlated with a low quality question. See my previous post about the Length of Conference Questions for more on this topic [3].

It seems to me that when running a lecture everyone involved has to agree about whether it’s a public lecture (IE one that is for any random people) as opposed to a society meeting (which while free for anyone to attend in the case of a LUG is for people with specific background knowledge). For a society meeting (for want of a better term) it’s OK to assume a minimum level of knowledge that rules out some people. If 5% of the audience of a LUG don’t understand a lecture that doesn’t necessarily mean it’s a bad lecture, sometimes it’s not possible to give a lecture that is easily understood by those with the least knowledge that also teaches the most experienced members of the audience.

For a public lecture the speaker has to give a talk for people with little background knowledge. Then the speaker and/or the MC have to discourage or reject questions that are for a higher level of knowledge.

As an example of how this might work consider the case of an introductory lecture about how an OS kernel works. When one of the experienced Linux kernel programmers visits Melbourne we could have an Eventbrite event organised for a lecture introducing the basic concepts of an OS kernel (with Linux as an example). At such a lecture any questions about more technical topics (such as specific issues related to compilers, drivers, etc) could be met with “we are having a meeting for more technical people at the Linux Users of Victoria meeting tomorrow night” or “we are having coffee at a nearby cafe afterwards and you can ask technical questions there”.

Planning Eventbrite Events

When experts in various areas of FOSS visit Melbourne they often offer a talk for LUV. For any such experts who read this post please note that most lectures at LUV meetings are by locals who can reschedule, so if you are only in town for a short time we can give you an opportunity to speak at short notice.

I would like to arrange to have some of those people give a talk aimed at a less experienced audience which we can promote through Eventbrite. The venue for LUV talks (Melbourne University 7PM on the first Tuesday of the month) might not work for all speakers so we need to find a sponsor for another venue.

I will contact Linux companies that are active in Melbourne and ask whether they would be prepared to sponsor the venue for such a talk. The fallback option would be to have such a lecture at a LUV meeting.

I will talk to some of the organisers of science and technology events advertised on Eventbrite and ask why they chose the times that they did. Maybe they have some insight into which times are best for getting an audience. Also I will probably get some idea of the best times by just attending many events and observing the attendance. I think that the aim of an Eventbrite event is to attract delegates who wouldn’t attend other meetings, so it is a priority to choose a suitable time and place.

Finally please note that while I am a member of the LUV committee I’m not representing LUV in this post. My aim is that community feedback on this post will help me plan such events. I will discuss this with the LUV committee after I get some comments here.

Please comment if you would like to give such a public lecture, attend such a lecture, or if you just have any general ideas.

LUV Hardware Library after 20 Months

20 months ago I started the LUV Hardware Library [1]. The aim of the project is to provide a repository of free spare parts for computers for the use of club members. People who have parts that are good but which they can’t use can donate them and others who need such parts can take them.

Some people have criticised my choice of the name “Hardware Library” because the word is associated with borrowing while with my Hardware Library it is expected that noone will return the item that they take. The Wikipedia page about libraries is worth reading, my interpretation of that is that the essential aspect of a library is that it is a public collection of items that are useful for study and that borrowing is just one thing that can be done. A book library could consist of a service of printing free books on demand (anyone could do this with access to The Gutenberg Project [2] and a printer) or of just making them available to download. Many libraries don’t allow books to be borrowed, they just allow them to be studied and copied in the library. Also every general public library has reference items that can’t be borrowed, it’s typical for a library to have a full encyclopedia which is not available to be borrowed. Also with the Hardware Library people feel obliged to give something if they take something (as happens with a geocache), so there is an issue of returning something.

My main aims with the Hardware Library were to save people money on parts and to help the environment by reducing the need to buy new computers when old ones can be upgraded and remain in service. My next aim was to help people learn about hardware by providing free parts, when a mistake has no financial cost people are more willing to experiment and will learn more. I believe that those aims have been achieved.

More Successful Than Expected

One thing that surprised me is the social aspect that developed. I had expected that most people would just find some parts that they need and not look at it again for some months. I had also anticipated that some people would poll the Hardware Library every month in the hope that a part they needed might appear. I didn’t expect that people would look through it every month because they just like looking at old hardware. I also didn’t expect groups of people to hang out by the Hardware Library to discuss various issues related to PC hardware and Linux.

During the breaks in the main meeting the location of the Hardware Library often becomes a focus for discussions of various issues related to Linux and hardware. I think that this is really advancing the aims of LUV [3] and I think that members of other LUGs should experiment with similar projects.

Starting this didn’t require any special skill or authority. I just started bringing a briefcase full of parts to meetings and offering them to whoever was interested. Any member of any LUG can do the same. To start something like this you wouldn’t even need a collection of parts, you could just bring a box and ask for donations.

Ownership of a Club

Last night at the Annual General Meeting we had a motion to disincorporate The Linux Users of Victoria (LUV) [1]. The proposal was for LUV to cease being an incorporated society on condition that Linux Australia (LA) [2] accepts us as a sub-committee. As a sub-committee of LA we would elect our own committee to run things locally but have LA hold the finances, deal with all the paperwork that the government demands, and generally do as many of the non-core tasks associated with running a users’ group as possible.

When we discussed this at the LUV committee meetings it didn’t seem like a big deal. But as is often the case with political discussions it turned out to be difficult.

There was a lot of discussion about LUV supposedly ceasing to exist, people seem to think that LUV is defined by having an incorporated society. My impression was always that it was defined by a mailing list and having meetings – and I was involved in both before there was an incorporated society.

Lurkers and Ownership

During the discussion we had some input from members who were typically lurkers who seemed to feel that their property rights towards LUV were being infringed, this annoys me. I think that if someone chooses not to be involved in running an organisation then they should choose not to concern themselves with the details of how the organisation is to be run. People who attend the meetings should have a say in how the meetings are run and have reason to be concerned about anything that might affect them and the opinions of speakers also matter. People who are involved with mailing list discussions should have a say in how the lists are run. But people who have never volunteered for a position on the committee shouldn’t be greatly concerned about the internal issues of how things are run.

Finance

Some concern was expressed about the financial situation of LUV and whether we would still get enough donations to keep it running when combined with LA. There was even some FUD suggesting that LA would just take our money (they had assured us that all funds and donations would be ear-marked for us). The current LUV financial situation is that Red Hat pays for the venue for the monthly meetings and the rent for the venue comprises about 2/3 of all donations. The remaining 1/3 comes from one company. So in the current situation if Red Hat ceased donating then we would have 18 months to find another donor or cease holding meetings before our bank balance became unreasonably low. If the company which gives the other significant annual donation was to cease doing so then we could operate for a few years on savings but we would need to find some other source of funding.

It seems to me that joining LA would give us more financial security. Then if Red Hat ceased paying for the venue then LA could keep things running until we found another donor, I’m confident that LA wouldn’t allow LUV to just shut down because of a shortage of donations.

If people are really concerned about the financial situation of LUV then they should urgently seek further donations such that if any one donor decided to stop giving then we could still operate as normal. To achieve that goal I think we need at least another $1,000 per annum. This issue of redundancy in donations is something I raise every time that LUV finances are discussed.

My conclusion is that people aren’t really bothered about the financial security of LUV except when they are looking for reasons to avoid change.

Doing New Things

During the course of discussion about the future of LUV there were a number of requests for improvement. One significant request was for more support for regional Linux users. Some years ago we held a mini-conference in Ballarat which went well. I think it would be good to do such things again, the cost is not particularly great and I’m sure it would be accepted by LA for funding, but we need to organise it.

Organising such events is something that anyone can do. Any LUV member can plan an event, get costs for everything that is needed (food, accommodation, travel, etc) and then pitch it to the LUV committee in terms of which things should be paid by LUV and which by the members concerned. We could then work on getting additional funding from LA if necessary. But planning an event takes some effort and it’s often effort that can only be done by a local. Finding a suitable venue and getting some assurance that a large enough audience will attend is something that can’t be done remotely.

I think that the problem for LUV in regard to such things isn’t a lack of money or independence. I think that the problem is that the committee spends too much volunteer time on administrative tasks and not enough time directly doing things that benefit members and the community in general.

In the past I have declined nomination to the LUV committee because I felt that I could contribute more by giving lectures, finding other speakers, and doing other things to directly improve the group. I was on the committee last year and have now been elected to it again, but I’m starting to think that I made a mistake. Maybe I should have declined and let others work on the new model rules and other paperwork.

One committee member has claimed that the time taken on administrative tasks isn’t taking time away from other LUV related tasks, I invite any committee members who feel that way to address some of the services that members are requesting. Speaking for myself my lack of time directly impacts that I can do for the club.

I think that ownership of a club should be related to what people do for the club. If you have a feeling of ownership and lack ideas for how to contribute then you can ask the LUV mailing list, there are lots of people with suggestions for things to do.

LUV Hardware Library

What is it?

Last month I started what I am calling the LUV Hardware Library. It’s a briefcase full of computer parts that are free to LUV members which I plan to bring to all LUV meetings [1]. The issue is that there is a lot of hardware which has no great value to people and is often excess to requirements but which is still somewhat expensive or difficult to obtain and thus transferring it from the people who don’t need it to people who do provides a significant benefit to the recipient but no real cost to the donor.

Currently my briefcase has a range of different types of DIMM, some PCI cards, some assorted cables, a laptop SATA disk, and a bunch of other random things. Most of the stuff has been in my spare parts pile for a year or two, but some of it has been donated by other people.

What we Need

The next thing we need is more packaging, anti-static bags for RAM and PCI cards, sealable bags for screws, and some way of storing motherboard batteries.

In terms of hardware that can be donated one thing that would be really useful is SATA disks. I’m sure that there are lots of people who have upgraded the storage of their computer and kept the old drive with no real chance of using it. But there are lots of people who need such disks, for example I’ve been giving away a bunch of P4 systems which lack disks and are not cabled for IDE disks – a supply of SATA disks would make them a lot more usable.

Also any other random electronic stuff is of interest, including non-Linux things such as mobile phones (but only if not network locked).

If you have something that’s big or heavy to donate then contact me via email first.

Other Options

Computerbank does great work in rebuilding old PCs and selling them cheaply to worthy people [2], but most of the spare hardware I get is below the minimum specs that they will accept. I’m not planning to compete with Computerbank in any way, I just want to provide a useful service to LUV members who want to upgrade their PC for free.

I encourage other people to do the same at other LUG meetings!