There’s a lot of advice about how to create and manage user passwords, and some of it is even good. But there doesn’t seem to be much advice about passwords for daemons, scripts, and other system processes.
I’m writing this post with some rough ideas about the topic, please let me know if you have any better ideas. Also I’m considering passwords and keys in a fairly broad sense, a private key for a HTTPS certificate has more in common with a password to access another server than most other data that a server might use. This also applies to SSH host secret keys, keys that are in ssh authorized_keys files, and other services too.
Table of Contents
Passwords in Memory
When SSL support for Apache was first released the standard practice was to have the SSL private key encrypted and require the sysadmin enter a password to start the daemon. This practice has mostly gone away, I would hope that would be due to people realising that it offers little value but it’s more likely that it’s just because it’s really annoying and doesn’t scale for cloud deployments.
If there was a benefit to having the password only in RAM (IE no readable file on disk) then there are options such as granting read access to the private key file only during startup. I have seen a web page recommending running “chmod 0” on the private key file after the daemon starts up.
I don’t believe that there is a real benefit to having a password only existing in RAM. Many exploits target the address space of the server process, Heartbleed is one well known bug that is still shipping in new products today which reads server memory for encryption keys. If you run a program that is vulnerable to Heartbleed then it’s SSL private key (and probably a lot of other application data) are vulnerable to attackers regardless of whether you needed to enter a password at daemon startup.
If you have an application or daemon that might need a password at any time then there’s usually no way of securely storing that password such that a compromise of that application or daemon can’t get the password. In theory you could have a proxy for the service in question which runs as a different user and manages the passwords.
Password Lifecycle
Ideally you would be able to replace passwords at any time. Any time a password is suspected to have been leaked then it should be replaced. That requires that you know where the password is used (both which applications and which configuration files used by those applications) and that you are able to change all programs that use it in a reasonable amount of time.
The first thing to do to achieve this is to have one password per application not one per use. For example if you have a database storing accounts used for a mail server then you would be tempted to have an outbound mail server such as Postfix and an IMAP server such as Dovecot both use the same password to access the database. The correct thing to do is to have one database account for the Dovecot and another for Postfix so if you need to change the password for one of them you don’t need to change passwords in two locations and restart two daemons at the same time. Another good option is to have Postfix talk to Dovecot for authenticating outbound mail, that means you only have a single configuration location for storing the password and also means that a security flaw in Postfix (or more likely a misconfiguration) couldn’t give access to the database server.
Passwords Used By Web Services
It’s very common to run web sites on Apache backed by database servers, so common that the acronym LAMP is widely used for Linux, Apache, Mysql, and PHP. In a typical LAMP installation you have multiple web sites running as the same user which by default can read each other’s configuration files. There are some solutions to this.
There is an Apache module mod_apparmor to use the Apparmor security system [1]. This allows changing to a specified Apparmor “hat” based on the URI or a specified hat for the virtual server. Each Apparmor hat is granted access to different files and therefore files that contain passwords for MySQL (or any other service) can be restricted on a per vhost basis. This only works with the prefork MPM.
There is also an Apache module mpm-itk which runs each vhost under a specified UID and GID [2]. This also allows protecting sites on the same server from each other. The ITK MPM is also based on the prefork MPM.
I’ve been thinking of writing a SE Linux MPM for Apache to do similar things. It would have to be based on prefork too. Maybe a change to mpm-itk to support SE Linux context as well as UID and GID.
Managing It All
Once the passwords are separated such that each service runs with minimum privileges you need to track and manage it all. At the simplest that needs a document listing where all of the passwords are used and how to change them. If you use a configuration management tool then that could manage the passwords. Here’s a list of tools to manage service passwords in tools like Ansible [3].
There is pwmd (https://gitlab.com/bjk/pwmd) that satisfies most, if not all, of these needs with the use of ACL’s for applications/users. But it doesn’t scale well. That shouldn’t be a problem for the uses you describe, though.
I think that the first question that should be asked when dealing with traditional service passwords is: do we need them? Is there any passwordless setup with equivalent or better security?
Here is a recent story from my job. We have a LDAP system where we keep all accounts that belong to human users. For technical reasons, we cannot delete accounts of former employees (lots of internal software would be unhappy with users disappearing), so we move them to a separate organizational unit, remove the password, remove them from important groups, set the login shell to /bin/false, and so on. I was given a task to write a script that reduces this to “move to the former-employees organizational unit, wait for the script to run from cron and to clean up everything else”.
The question is: which account should this script use to bind to the LDAP server? How to store the password? Who should know it? The boss is really worried about soon-to-be-former employees stealing passwords from service accounts, and this script has quite a lot of access to be a valuable target if its password is stolen.
The solution that we ended up with was to run the script directly on the LDAP server, as a separate local UNIX user, from cron, connect to the LDAP server via ldapi:/// and use the EXTERNAL SASL mechanism. This way, authentication is done based solely on the ability to connect to a UNIX-domain socket and to present the correct UID and GID. Permissions are granted to dn.exact=”gidNumber=917+uidNumber=917,cn=peercred,cn=external,cn=auth”. No passwords are needed, nothing to steal, no way to assume the script’s identity without becoming local root on the LDAP server first.
Ben: while pwmd seems useful for storing data, there’s still the issue of how to determine that only the correct application can access it.
One thing that just occurred to me is that if you have a centralised password store you could log which apps connect to it and then know which accounts are in use. Not all services make it easy to determine whether an account is used and if you have dozens of services it would be difficult to track them all.
Alexander: yes Unix domain sockets remove some of the issues. But that’s only for local access. Also for LDAP in some situations you can remove application access by binding to the LDAP server with the client’s credentials. One time I setup a RADIUS server that bound to the LDAP server with the client’s username and password so there was no LDAP account for the RADIUS server that could be stolen.
etbe: In pwmd each XML password element can be protected with an ACL. The ACL is compared to the current client which may be a username or TLS fingerprint in the case of a remote one.