On the 1st of August I noticed that the server which runs my blog among other things was having an occasional SEGV from a sshd process. Unfortunately I was busy and didn’t pay much attention to this, which turned out to be a big mistake.
On the 12th of September I started investigating this properly and noticed that when someone tried to connect to ssh with password authentication sshd would SEGV after it was denied access to a shared memory region or a semaphore which had a SE Linux type of unconfined_t. I added some SE Linux auditallow rules and discovered that the memory region in question was created by the ssh client. Shortly after that I came to the conclusion that this wasn’t some strange feature of ssh (or one of the many shared objects it uses) but hostile activity. The ssh client appeared to be storing passwords that it used in a shared memory region and sshd was also collecting passwords in the same region and presumably offering them to a ssh client which uses some extension to the ssh protocol.
The sshd process was crashing because it couldn’t handle EPERM on access to shared memory or semaphores. Presumably if the system in question wasn’t running SE Linux then the exploit would have remained undetected for a lot longer.
At this stage we don’t know how the attacker got in. Presumably one of the people with root access ran a ssh client on a compromised system and had their password sniffed. One such client system was mysteriously reinstalled at about that time, the sysadmin of the system in question claimed to have no backups which made it impossible to determine if that system had been compromised. I believe that the sysadmin of the client system knew that their system was compromised, kept that information secret, and allowed other systems to become and remain compromised.
The attacker made no good effort to conceal their presence, they replaced ssh, sshd, and ssh-add and didn’t bother changing the Debian checksums so the debsums program flagged the files as modified. Note that I have kept copies of the files in question and am willing to share them with anyone who wants to analyse them.
Steinar H. Gunderson has named this trojan Ebury [1].
Table of Contents
Recovery
By the evening of the 13th of September I had the system mostly working again. Jabber still isn’t working because ejabberd is difficult to get working at the best of times, I am now investigating whether there is a better Jabber server to use, but as I don’t use Jabber often this hasn’t been a priority for me.
Some of the WordPress plugins I use and all of the WordPress themes that are installed were outside the Debian packaging system, as I couldn’t be sure that they hadn’t been altered (because the people who wrote WordPress plugins don’t keep old versions online) I had to upgrade to the newer versions. Of course the newer versions weren’t entirely compatible so I had to use a different theme and I couldn’t get all plugins working. Link Within no longer works, not that it ever worked properly [2], I wanted to try Outbrain again but their web site won’t let me login (and they haven’t responded to my support request). Does anyone know of a good WordPress plugin to provide links to related content? Either related content on my blog or on the Internet in general will be OK.
Some people have asked me about the change in appearance of my blog. It was simply impossible (for someone with my PHP skills) to get my blog looking the same way as it did before the server was cracked. I think that the new look is OK and don’t mind if people think it looks likw a VW advert – VW make great cars, I was very satisfied with the VW Passat I used to drive.
Future Plans
I had bought some Yubikeys (USB devices that generate one-time passwords) [3] to control access to that server, if I had configured the software to use them then this might not have happened. The use of one-time password devices can prevent passive password-sniffing attacks. It would still allow active attacks (such as using ControlPath/ControlMaster options on the ssh client to allow a hostile party to connect later (EG the -M, -S, and “-o ControlPersist” options for the ssh client). It’s a pity that there doesn’t seem to be a way to configure the ssh server to disable ControlMaster.
Conclusion
It would be good to have some changes to sshd to allow more restrictions on what a client can request, as ControlMaster functionality isn’t needed by most users it should be possible to disable it.
SE Linux doesn’t protect against a compromised client system or any other way of stealing passwords. It did do a good job of stopping Ebury from doing all the things it wanted to do and thus making me aware of the problem. So I count this as a win for SE Linux.
Yubikeys are the cheapest and easiest way of managine one-time passwords. I had already bought some for use on the system in question but hadn’t got around to configuring them. I have to make that a priority.
Another option you might consider for OTP is using “google authenticator” (http://itand.me/google-authenticator-for-easy-one-time-passwo , it’s has been packaged for debian ). The only hardware requirement is a smartphone.
I maintain ejabberd in Debian, so if you have a problem, try to describe it and may be I’ll be able to help. You can drop me a message (my e-mail address is attached to this post) or I’m reachable via xmpp as kostix@007spb.ru.
Also it’s okay to post your problem’s description to the ejabberd mailing list (it’s English-speaking).
Hey, Russell, did you try http://wordpress.org/extend/plugins/yet-another-related-posts-plugin/ – this is “the only one” related post plugin for wordpress, try it, usually just works out of the box and is easy to configure and not so demanding – the author seems to care about sql optimization – this is something special among wp devs ;)
Regarding your layout problems – I can help you easily, just send me a ssh login to your server so I can
messset up everything… ok, that was a joke, obviously :)A very good and easy theme for own customizations can be found here: http://www.rootstheme.com/ – will make your blog look good ootb.
BTW wouldn´t you like to write anything about Android + SELinux? Is there any chance of an easily usable distribution of selinux-enhanced android, maybe via custom rom images like e.g. cyanogenmod? Would be great to have more open source security on mobile devices!
Thank you for your interesting articles and good luck for 2012!
Thanks for the suggestion Bughunter, it’s early days but YARPP seems to be working well for me. The Roots gallery looks nice, but I’ll delay theme changes for a while, it’s painful.
As for Android + SE Linux, nothing about Android is easily usable unless you just use what your Telco gives you. If you want to mod your phone then it’s a lot of pain. Given that SE Linux on it’s own has been regarded as too much effort for so many people it seems unlikely that SE-Android is going to become popular.
Besides the best thing that could be done for Android security would be to make it easy to restrict the permissions of applications. Ideally the Android market would allow you to specify which permissions are acceptable at search time (EG I don’t want games that want permission to make phone calls) and the environment would permit running an application with less permissions than it requests (something that CyanogenMod can do). If we had SE Linux lock down an Android system and then still permit rogue apps to read/write all stored data and make phone calls then we probably wouldn’t gain much.
To summarise, SE Linux only helps if your system is basically secure to start with, and that involves being used in a secure manner.
Konstantin: Thanks for that offer, and I hope you didn’t take this article as criticising your work. My problems are with the implementation of Erlang and some design choices made by eJabberd upstream. I will get back to eJabberd, but it may be some time.
Luca: Thanks for the suggestion, but I would prefer to have Google not entirely own me. I would prefer a challenge-response algorithm calculated by an Android app to something that relies on the Google cloud. But for use with a desktop or laptop system the Yubikey is by far the easiest option.
You seem to misunderstand the Google Authenticator. It is “just” an implementation of TOTP/HOTP. The Android App does not request any permissions at all (okay, it want to be able to use the vibration motor). See http://code.google.com/p/google-authenticator/ for all the code. No component of the framework needs any connection to Google at all.
Tobias: Thanks for that correction. I had heard about Google doing authentication by SMS and had got the idea that was what Luca was referring to, I really should have read the link he provided.
What you describe sounds very useful and I’ll have to try it out.
About SELinux: in the wiki page I see it is developed against aosp master branch – is there any chance getting this backported to some older branch, say Gingerbread? CyanogenMod + SELinux certainly would be the best combination – with the apps market you are right, but I was thinking about people that *want* a safe phone, not the ones that install every game app :) – I believe SELinux would definitely make sense for many, but it should come in an “easy” interface, maybe as a configuration option of CyanogenMod – that way more people could adapt it. As in the pc world of course it will take some time to make people understand the consequences of not really owning the hardware you bought…
I might be mistaken, but sshd’s MaxSessions set to 1 will practically disable the ControlMaster option.