It has recently been announced that Debian had a serious bug in the OpenSSL code [1], the most visible affect of this is compromising SSH keys – but it can also affect VPN and HTTPS keys. Erich Schubert was one of the first people to point out the true horror of the problem, only 2^15 different keys can be created [2]. It should not be difficult for an attacker to generate 2^15 host keys to try all combinations for decrypting a login session. It should also be possible to make up to 2^15 attempts to login to a session remotely if an attacker believes that an authorized key was being used – that would take less than an hour at a rate of 10 attempts per second (which is possible with modern net connections) and could be done in a day if the server was connected to the net by a modem.
John Goerzen has some insightful thoughts about the issue [3]. I recommend reading his post. One point he makes is that the person who made the mistake in question should not be lynched. One thing I think we should keep in mind is the fact that people tend to be more careful after they have made mistakes, I expect that anyone who makes a mistake in such a public way which impacts so many people will be very careful for a long time…
Steinar H. Gunderson analyses the maths in relation to DSA keys, it seems that if a DSA key is ever used with a bad RNG then it can be cracked by someone who sniffs the network [4]. It seems that it is safest to just not use DSA to avoid this risk. Another issue is that if a client supports multiple host keys (ssh version 2 can use three different key types, one for the ssh1 protocol, one for ssh2 with RSA, and one for ssh2 with DSA) then a man in the middle attack can be implemented by forcing a client to use a different key type – see Stealth’s article in Phrack for the details [5]. So it seems that we should remove support for anything other than SSHv2 with RSA keys.
To remove such support from the ssh server edit /etc/ssh/sshd_config and make sure it has a line with “Protocol 2“, and that the only HostKey line references an RSA key. To remove it from the ssh client (the important thing) edit /etc/ssh/ssh_config and make sure that it has something like the following:
Host *
Protocol 2
HostKeyAlgorithms ssh-rsa
ForwardX11 no
ForwardX11Trusted no
You can override this for different machines. So if you have a machine that uses DSA only then it would be easy to add a section:
Host strange-machine
Protocol 2
HostKeyAlgorithms ssh-dsa
So making the default configuration of the ssh client on all machines you manage has the potential to dramatically reduce the incidence of MITM attacks from the less knowledgable users.
When skilled users who do not have root access need to change things they can always edit the file ~/.ssh/config (which has the same syntax as /etc/ssh/ssh_config) or they can use command-line options to override it. The command ssh -o “HostKeyAlgorithms ssh-dsa” user@server will force the use of DSA encryption even if the configuration file requests RSA.
Enrico Zini describes how to use ssh-keygen to get the fingerprint of the host key [6]. One thing I have learned from comments on this post is how to get a fingerprint from a known hosts file. A common situation is that machine A has a known hosts file with an entry for machine B. I want to get the right key in machine C and there is no way of directly communicating between machine A and machine C (EG they are in different locations with no network access). In that situation the command “ssh-keygen -l -f ~/.ssh/known_hosts” can be used to display all the fingerprints of hosts that you have connected to in the past, then it’s a simple matter of grepping the output.
Docunext has an interesting post about ways of mitigating such problems [7]. One thing that they suggest is using fail2ban to block IP addresses that appear to be trying to do brute-force attacks. It’s unfortunate that the version of fail2ban in Debian uses /tmp/fail2ban.sock for it’s Unix domain socket for talking to the server (the version in Unstable uses /var/run/fail2ban/fail2ban.sock). They also mention patching network drivers to add entropy to the kernel random number generator. One thing that seems interesting is the package randomsound (currently in Debian/Unstable) which takes ALSA sound input as a source of entropy, note that you don’t need to have any sound input device connected.
When considering fail2ban and similar things, it’s probably best to start by restricting the number of machines which can connect to your SSH server. Firstly if you put it on a non-default port then it’ll take some brute-force to find it. This will waste some of the attacker’s time and also make the less persistent attackers go elsewhere. One thing that I am considering is having a few unused ports configured such that any IP address which connects to them gets added to my NetFilter configuration – if you connect to such ports then you can’t connect to any other ports for a week (or until the list becomes too full). So if for example I had port N configured in such a manner and port N+100 used for ssh listening then it’s likely that someone who port-scans my server would be blocked before they even discovered the SSH server. Does anyone know of free software to do this?
The next thing to consider is which IP addresses may connect. If you were to allow all the IP addresses from all the major ISPs in your country to connect to your server then it would still be a small fraction of the IP address space. Sure attackers could use machines that they already cracked in your country to launch their attacks, but they would have to guess that you had such a defense in place, and even so it would be an inconvenience for them. You don’t necessarily need to have a perfect defense, you only need to make the effort to reward ratio be worse for attacking you than for attacking someone else. Note that I am not advocating taking a minimalist approach to security, merely noting that even a small increment in the strength of your defenses can make a significant difference to the risk you face.
Update: based on comments I’m now considering knockd to open ports on demand. The upstream site for knockd is here [8], and some documentation on setting it up in Debian is here [9]. The concept of knockd is that you make connections to a series of ports which act as a password for changing the firewall rules. An attacker who doesn’t know those port numbers won’t be able to connect. Of course anyone who can sniff your network will discover the ports soon enough, but I guess you can always login and change the port numbers once knockd has let you in.
Also thanks to Helmut for advice on ssh-keygen.
- [1] http://www.debian.org/security/2008/dsa-1571
- [2] http://blog.drinsama.de/erich/en/linux/2008051401-consequences-of-sslssh-weakness.html
- [3] http://changelog.complete.org/posts/714-Thoughtfulness-on-the-OpenSSL-bug.html
- [4] http://blog.sesse.net/blog/tech/2008-05-14-17-21_some_maths.html
- [5] http://www.phrack.org/issues.html?id=11&issue=59
- [6] http://www.enricozini.org/2008/tips/ssh-host-key-fingerprint.html
- [7] http://www.docunext.com/blog/2008/05/14/my-security/
- [8] http://www.zeroflux.org/cgi-bin/cvstrac.cgi/knock/wiki
- [9] http://www.ducea.com/2006/07/05/how-to-safely-connect-from-anywhere-to-your-closed-linux-firewall/
I had pondered the same things as in your final paragraph in the past. After using a similar approach for more than a year, I eventually decided against it and went with a more-or-less opposite methodology. It may be suitable for you too.
The initial approach was implemented using fail2ban (to detect SSH brute-force attacks), knockd (for port scanning checking), and a custom iptables script (to do some extra stuff when adding and removing entries from filter chains). My iptables set-up was quite elaborate, with blackholes for various subnets (which blocked out much of China and other brute-forcing countries), and a grand list of brute-force offenders. Like your idea, they were blocked for a significant duration of time (I think my approach was for one month), with exponential extensions if they were repeat offenders.
My current approach is to keep using port 22 for SSH, but instead block all but a few specific IPs to connect. This eliminates the brute-forcing attempts entirely. To allow connections from an unknown IP, I still use knockd; only this time it allows new connections to port 22 from the knocker for 1 minute (after poking a sequence of ports).
By the time I changed methodologies, I had ~600 IPs on my block list, 12 class-C subnets (manually entered by analyzing the blocked IP list), and ~1200 IPs in the “archived” block list. This was a pretty bad performance hit.
Hi Russell,
The MetaSploit project has already generated an exhaustive list of 1024-bit DSA keys and all 1023, 1024, 2047 and 4096 bit RSA keys. They also generated all 8192-bit RSA keys for PID’s of up to 4100 for good measure.
For the smaller keys it didn’t take too long, they write:
It’s also worth noting the the patch was posted first to the openssl-dev list before being applied to the Debian package, and got this response:
Apparently, though, openssl-dev isn’t a list for development of OpenSSL, according to at least one core developer, to which Branden Robinson has blogged this witty response..
Pretty good way to secure SSH access is to block it on firewall, and use knockd to open SSH access for given time and for given host after specified combination of packets (from this host). For example SYN packet on ports 1000, 2000, 3000, 4000. More on: http://www.ducea.com/2006/07/05/how-to-safely-connect-from-anywhere-to-your-closed-linux-firewall/
Package is included in Debian, work very well. Not recommended on weak machines with lots of traffic (over 20 mbit).
The known_hosts file merely is the concatenation of public keys. As we have learned before public keys can be inspected using ssh-keygen -l -f. Thus the desired fingerprints can be displayed using:
ssh-keygen -l -f ~/.ssh/known_hosts
Easy, isn’t it?
PS: ssh-vulnkey works the very same way: it inspects public keys. Therefore ssh-vulnkey ~/.ssh/known_hosts | grep COMPROMISED may also be interesting. On my host this has the following output for instance:
COMPROMISED: 2048 ca:5d:c6:00:ac:3b:4a:ef:61:46:1c:ef:b3:c9:d3:6d play.coker.com.au,220.237.149.31
To block port scan, you can try to use recent match in netfilter (IPv4 only).
http://www.zeroflux.org/cgi-bin/cvstrac.cgi/knock/wiki
T Lin and rozie: I had heard about knockd and similar systems (I included what seems to be the main URL above to save people the Google effort), it does seem appealing and you do have a point about it being easier to manage. I’ll have to consider that.
Chris: I had noticed that page earlier, as far as I recall it didn’t have such a comprehensive set of keys last time I checked.
As for the patch, I’m writing another blog post which will mention that issue.
You could use portsentry to block hosts which scan you. It can be told to monitor a list of specific ports, or to monitor all unused ports. In the latter mode, you can give it a list of ports to ignore, and in either mode you can give it a list of source addresses to ignore (e.g. your LAN).
You’d need to write a script to scan portsentry’s log and unblock addresses based on whatever criteria you choose, but since portsentry logs all blocked hosts with a timestamp for each, that’s not hard to do.
fail2ban is a very nice piece of software, and I’ve made frequent use of it in the past (and still do).
iptables provides some similar capabilities, too. There’s two different ways of protecting against brute force ssh attacks with iptables.
First, if the rate of NEW connections to port 22 (or whatever port you’re running ssh on) exceeds a specified hitcount, drop them. For example, to drop NEW connections if there is more than 6 SYN’s sent in 180 seconds:
-A INPUT -p tcp -m tcp –dport 22 -m state –state NEW -m recent –set –name abusers –rsource
-A INPUT -p tcp -m tcp –dport 22 -m state –state NEW -m recent –update –seconds 180 –hitcount 6 –name abusers –rsource -j DROP
Alternately, you can use iptables to only accept NEW connections at a certain rate. Any host that exceeds this rate no longer qualifies for the ACCEPT, and will fall through to the DROP/REJECT that of course you have following it (either explicitly for ssh, or your final rule, because deny by default is good practice). To only accept NEW connections at a rate of 1 per minute, with an initial burst of 4, you can do this:
-A INPUT -p tcp -m tcp –dport 22 -m state –state NEW -m limit –limit 1/min –limit-burst 4 -j ACCEPT
I’ve had very good luck with both styles of rate limiting. Just be careful not to set things *too* agressive, as I’ve seen some cases where multipl SYNs are sent very rapidly when a program is establishing a connection.
I’ve been using portsentry for a year or so, and one problem is that internet is full of viruses scanning the net from hosts with dynamic ip: hosts.deny becomes quite long shortly, but the content becomes useless shortly either. The same apply to iptables rules.
I’m using a dynamic ip, too, and bypassed the problem restarting the connection at some hour during the night, if hosts.deny reached a certain number of entry (well, one could do it every night regardless hosts.deny…): after changing ip I can safely wipe hosts.deny and all added iptables rules
obviously portsentry cannot block brute force attack to the port 22, so you should move ssh to some other port in order to make it useful in this sense
on a static ip, I don’t think portsentry could be a valuable solution: one could scan a certain number of ports before being blocked, then change ip and restart scanning, or one could sniff some traffic and see which ports are used
regards
tindal
I’ve heard good things about portknock, but for some reason never tried it. I’m curious about how it will work for you.
And thanks for mentioning randomsound. I’m running audio-entropyd on one of my machines and have had good experiences with it:
http://www.vanheusden.com/aed/
> So if for example I had port N configured in such a manner and port N+100 used for ssh listening then it’s likely that someone who port-scans my server would be blocked before they even discovered the SSH server.
Hmm, wouldn’t that allow me to deny *you* access to your own server if I knew your IP address (and are able to spoof it)?
All additional security issues are kinda nice, but what’s really needed is a good default configuration. Most administrators and users will not apply additional security measures.
Port knocking is effectively just a shared secret. Can’t the SSH protocol be improved to incorporate something like that?
> The next thing to consider is which IP addresses may connect. If you were to allow all the IP addresses from all the major ISPs in your country to connect to your server then it would still be a small fraction of the IP address space.
And get locked out of your own server if for whatever reason you get assigned an address out of this range?
Olaf@11: Yes. But any mechanism that attempts to recognise bad behavior and lock out the perp will occasionally do that.
Systems such as port knocking are based around the idea that sshd might somehow fail. If you make that assumption then it’s best not to have sshd be part of the solution.
Olad@12: Yes. But in many cases this is an acceptable trade-off.
[…] Comments etbe on Debian SSH Problemspa on Software Development is a Team SportAndre Felipe Machado on Ideas to Copy from Red HatOlaf van […]