I think that a major deficiency in Computer Science degrees is the lack of sysadmin training.
Table of Contents
Version Control
The first thing that needs to be added is the basics of version control. CVS (which is now regarded as obsolete) was initially released when I was in the first year of university. But SCCS and RCS had been in use for some time. I think that the people who designed my course were remiss in not adding any mention of version control (not even strategies for saving old versions of your work), one could say that they taught us about version control by letting us accidentally delete our assignments. :-#
If a course is aimed at just teaching programmers (as most CS degrees are) then version control for group assignments should be a standard part of the course. Having some marks allocated for the quality of comments in the commit log would also be good.
A modern CS degree should cover distributed version control, that means covering Git as it’s the most popular distributed version control system nowadays.
For people who want to work as sysadmins (as opposed to developers who run their own PCs) a course should have an optional subject for version control of an entire system. That includes tools like etckeeper for version control of system configuration and tools like Puppet for automated configuration and system maintenance.
Dependencies
It’s quite reasonable for a CS degree to provide simplified problems for the students to solve so they can concentrate on one task. But in the real world the problems are more complex. One of the more difficult parts of managing real systems is dependencies. You have issues of header files etc at compile time and library versions at deployment. Often you need a program to run on systems with different versions of the OS which means making it compile for both and deal with differences in behaviour.
There are lots of hacky things that people do to deal with dependencies in systems. People link compiled programs statically, install custom versions of interpreters in user home directories or /usr/local for daemons, and do many other things. These things can have bad consequences including data loss, system downtime, and security problems. It’s not always wrong to do such things, but it’s something that should only be done with knowledge of the potential consequences and a plan for mitigating them. A CS degree should teach the potential advantages and disadvantages of these options to allow graduates to make informed decisions.
Backups
I’ve met many people who call themselves computer professionals and think that backups aren’t needed. I’ve seen production systems that were designed in a way that backups were impossible. The lack of backups is a serious problem for the entire industry.
Some lectures about backups could be part of a version control subject in a general CS degree. For a degree that majors in Sysadmin at least one subject about backups is appropriate.
For any backup (even backing up your home PC) you should have offsite backups to deal with fire damage, multiple backups of different ages (especially important now that encryption malware is a serious threat), and a plan for how fast you can restore things.
The most common use of backups is to deal with the case of deleting the wrong file. Unfortunately this case seems to be the most rarely mentioned.
Another common situation that should be covered is a configuration error that results in a system that won’t boot correctly. It’s a very common problem and one that can be solved quickly if you are prepared but which can take a long time if you aren’t.
For a Sysadmin course it is important to cover backups of systems in remote datacenters.
Hardware
A good CS degree should cover the process of selecting suitable hardware. Programmers often get to advise on the hardware used to run their code, especially at smaller companies. Reliability features such as RAID, ECC RAM, and clustering should be covered.
Planning for upgrades is a very important part of this which is usually not taught. Not only do you need to plan for an upgrade without much downtime or cost but you also need to plan for what upgrades are possible. Next year will your system require hardware that is more powerful than you can buy next year? If so you need to plan for a cluster now.
For a Sysadmin course some training about selecting cloud providers and remote datacenter hosting should be provided. There are many complex issues that determine whether it’s most appropriate to use a cloud service, hosted virtual machines, hosted physical servers managed by the ISP, hosted physical servers purchased by the client, or on-site servers. Often a large system will involve 2 or more of those options, even some small companies use 3 or more of those options to try and provide the performance and reliability they need at a price they can afford.
We Need Sysadmin Degrees
Covering the basic coding skills takes a lot of time. I don’t think we can reasonably expect a CS degree to cover all that and also give good coverage to sysadmin work. While some basic sysadmin skills are needed by every programmer I think we need to have separate majors for people who want a career in system administration.
Sysadmins need some programming skills, but that’s mostly scripting and basic debugging. Someone who’s main job is as a sysadmin can probably expect to never make any significant change to a program that’s more than 10,000 lines long. A large amount of the programming in a CS degree can be replaced by “file a bug report” for a sysadmin degree.
This doesn’t mean that sysadmins shouldn’t be doing software development or that they aren’t good at it. One noteworthy fact is that it appears that the most common job among developers of the Debian distribution of Linux is System Administration. Developing an OS involves some of the most intensive and demanding programming. But I think that more than a few people who do such work would have skipped a couple of programming subjects in favour of sysadmin subjects if they were given a choice.
Suggestions
Did I miss anything? What other sysadmin skills should be taught in a CS degree?
Do any universities teach these things now? If so please name them in the comments, it is good to help people find universities that teach them what they want to learn and help them in their career.
FYI, I replied to your post in http://www.lucas-nussbaum.net/blog/?p=912
Usenix spent some effort on this topic, see for example https://www.usenix.org/jesa
I disagree, and don’t think this should be part of CS.
Computer Science is exactly what it says: the science (and mathematics) of computation theory. It is a very wide subject from semantics and discrete number theory, through proof and correctness, boolean algebra and semiconductor technology, to set-relational mathematics, information theory and data-structures and algorithms.
Teaching someone how to be a sysadmin or a software developer or a network operator is more akin to an engineering degree. I’d suggest we keep CS a “science” – but make sure that CS students do e.g. a group and an individual final-year project that makes them have to apply their academic skills to learning some of the real-world stuff.
Marek: Computer Science degrees are mostly undertaken by people who plan to work in a corporate IT environment. While they do have some hard science it’s often a lot less than you might desire, I guess it depends on which university you attend and which options you choose. I agree that sysadmin is more like Engineering, but the terms “Computer Science” and “Software Engineering” are used fairly interchangeably so I think that issue is covered.
If you think that a university should offer 2 separate courses named “Computer Science” and “Software Engineering” and only have sysadmin stuff in the second then I think it sounds like a reasonable way of doing things.
That said I recently sat in on first-year Engineering lectures and they had more of a mathematical and scientific approach than the Computer Science lectures I recall from the 90’s. I’m pretty sure that most Engineering is based firmly on scientific principles.
name: JESA sounds like a good project.
Lucas: It’s very rare for companies to hire recent graduates for sysadmin work because none of the courses cover it and sysadmins mostly learn on the job. Hire a graduate as a programmer and they have had some relevant training, hire a graduate as a sysadmin and they have had none.
I agree that version control probably isn’t one of the more important sysadmin skills. The order of items in this post is mostly based on their relevant to core Computer Science. Version control is essential to every programmer even if they have no interest in sysadmin.
Dependencies are something that you inevitably encounter when doing a serious software development project. When I was at university the major final year project was in a group of about 12 students (which is larger than most software development teams) but had no use of external libraries and no talking to external systems. The project was supposed to be similar to real world software development, but whenever you have a real world project with 12 programmers you end up talking to databases, web services, and all sorts of other things.
I don’t think that dependencies and backups are 2 of the most important things to be taught in university for a CS or SE degree. But I do think that they are 2 of the most significant omissions from what I know of current courses.
In the old days it was expected that every Unix sysadmin was also a capable systems programmer. Nowadays that’s not a requirement and I think this is generally a good thing. It’s similar to how most drivers were capable mechanics in the early days of cars.
It shouldn’t be the case that a small company with a single person doing IT work should have to either pay the type of salary that might make you or I interested in such a job or have someone who can’t do it properly. Hiring someone without much skill at programming to do 99% of the work and hiring someone with specific skills on an hourly basis for the other 1% is a more effective way of running things.
I think that a major success of the Debian project (along with other distributions such as Fedora) is in making most sysadmin work possible without such expert skills.
Lucas I’m not seeing a great disagreement between us here. It seems to be that we both have the same beliefs about which skills should be taught, we have different observations about what is actually being taught, and some subtle disagreements about the details of teaching.
I don’t claim to know everything about what is being taught at university. I am basing my article on what I was taught and by my observations of colleagues demonstrating a lack of knowledge in areas that I think should be taught at university.
I’m not really set on the details of how much time should be spent in different areas. Version control and backups could each be given a basic coverage in a few lectures and a few practical sessions. Students who want to learn more could do optional subjects.
When I went to university business knowledge was the fad. Even though I deliberately chose the course that had the least business subjects I still had to learn accounting and economics. I think that substituting backups and version control subjects for accounting and economics would make everyone in this discussion happier. ;)
I think git-based version control is an essential first year undergrad CS topic, we certainly cover that almost on day one here at Imperial College London. We also have an in-house gitlab server, which we bulk create individual student and exercise repos on, as well as using it for normal github-style creation of personal and group repos for projects via the web interface. But that’s not specific to sysadmins, all CS students should be familiar with git these days.
On the wider “teaching systems admin” points, I’m not sure whether systems admin could/should fit “within” a CS degree, or be an alternative to it. My suggestion would be that the place to start would be to create an individual lecture course covering some of the key issues in systems admin, probably an optional course aimed at third/fourth year students. Has anyone done that? we haven’t:-)
For me the key overarching concern of large-scale systems admin is automation, especially automated installation and configuration management, whether using puppet or cfengine or home-brew solutions (we do the latter). Certainly, building software (and packaging software, and releasing packages to sets of machines) are important too, that’s going to get into the software dependencies problem too.
Only final thing: re: the question of whether CS is Science or Engineering? I’m firmly of the opinion that the whole subject is an Engineering discipline, not a Science. That’s certainly our approach. Plus, I have a personal rule: any subject that needs to include “Science” explicitly in the name, probably isn’t:-)
examples: Political Science, Computing Science, Sports Science, Nutrition Science.
The typical (Unix) sysadmin guy I picture seems to be aging along with his habitat. Slowly this habitat is changing and moving. The growth of Linux keeps him occupied for now. But just like our world is getting bigger (not smaller) and we feel pressed to have educated opinions about world politics and problems the IT world is also growing. Like Russell said, dependencies are getting more complex. Think about how the number of packages in Centos has grown. But modern dependencies are not only on the OS level. There is an abstraction going on with both more layering and physical and operational interdependencies with several on and off premise (AAS) technologies, suppliers and contractors. New storage options pop up like NoSQL clusters and block storage and besides we have to grasp how to manager containerization properly. Of course there is config management tooling available to manage the infrastructure but this requires quite some specialization and effort.
The piece by Thomas Limoncelli sums it up pretty well. https://www.usenix.org/system/files/jesa/jesa_0101_limoncelli.pdf
He characterizes the change in the field of system administration as going from assembling cars to developing and managing the robots that build them. Besides these sky high requirements (is there still a place for system operators besides administrators?) the following quote struck me:
“It is tempting to focus the curriculum on building ideal systems from the ground up. However, the typical sysadmin’s
job is the opposite: they jump into an existing system which others have built, then slowly evolve it over time. Only
once in my 25 years in system administration have I had the opportunity to start from a green field.”
So systems engineering should at least pay attention to the process aspects, how to evolve over time and do proper life cycle management.
Duncan: Thanks for your comment, it sounds like Imperial College is doing some good things.
Interesting comment about science in the name. ;)
Richard: Thanks for the comment, haven’t been in contact with you for ages!
The changing habitat thing is interesting. Friends have been telling me about some of the changes, some good and some bad. I’m not about to blog about this in the near future as I’ve been mostly stuck in the old habitat and haven’t had personal experience of the changes. Maybe it would be a good topic for you to blog about.
The article by Tom Limoncelli is excellent (as usual for his work), everyone read that!
Thanks Russell, really interesting blog post. Oh, I missed one obvious “non-science”: Creation Science. See how it works:-)
I’d be very interested in knowing how many universities have tried teaching a systems admin lecture course, within a traditional CS degree, and how well that went.
Richard also makes some excellent points, I quite agree with the Thomas Limoncelli article – especially about the importance of systems administrators being programmers too, and using a “ruthless automation” approach to their systems admin work. I’ve never been quite certain what “DevOps” means, but if it means the application of programming and ruthless automation to large scale systems administration, then I’m in complete agreement. Personally I describe myself as a “systems programmer”, perhaps it’s time for that old term to stage a comeback:-)
One other point that cropped up in a personal discussion: one characteristic that sysadmins need, that I suspect can’t be taught, is the ability to handle interrupts (or “questions from users” as they’re otherwise known) well. I’ve known several gifted software developers who transferred to systems administration and got really stressed by the constant low-level of interruptions. Obviously “on call” rotas can help – by funnelling interruptions to people who know that they will be interrupted for the next N hours, with the implication that the rest of the sysadmins (those not on call right now) can concentrate more deeply. Personally, I’ve had quite a high tolerance for interrupts throughout my career, so don’t find that anywhere near so stressful. But that’s not by my conscious doing – it’s just my character I guess.
While I agree that CS degrees should include CVS knowledge, I don’t agree that addition of that knowledge to the curriculum is going to produce sys admins.
I regard a degree as evidence that someone has the foundation on which to build skills (a basic understanding of systems and methodology, good research skills, and the ability to write). For certification of sys admin skills I rate LPI/RedHat or TAFE qualifications over a CS degree.
I have come across a few “veteran” sys admins that don’t know CVS, but the most common failings (IMO) are problem solving (isolation testing and *proof* of a solution), time management, proper use of email (it’s not a #$%^@ electronic postcard system!), and documentation.
To be fair I rarely get to work on systems where the administrators “had a clue”. Usually they “learn” solutions from Google rather than system documentation, debugging, and reading logs (forums are not the font of knowledge and StackExchange is no substitute for man pages??).
As for DevOps… surely that belongs in the same bucket as Daylight Saving and 12 hour rotating shifts?
Correction: The current ICT50315 Diploma of Information Technology System Administration doesn’t appear to include version control… (or backups, except as part of Plan and Manage Virtual Computing). Surely even in the Microsoft-centric TAFE world they’ve heard of these things!
So much for my expectation that TAFE graduates should be able to hit the ground running (I was wrong).