Puppet

One more comment: The alternative solution is not so good protection as a potential attacker who gained puppet user privileges could still read/copy the private key from disk and thereby produce correct authentication. Otherwise he'd have to access memory in some way which is more difficult (and AFAIK inevitable).

A tentative patch... I am a puppet newbie, so no idea what the side-effects are, not really tested neither. ;-)

Luke - I have some ideas on this but it is:

a) Likely to be impacted by past history - I looked at the perm. change on rundir in the patch and sighed.
b) Likely to be/been refactored in 0.25.x
c) Likely to provoke some strong opinions and its late.

Thoughts?

Fixed two bugs. For me this works now.

The puppetmasterd process creates the CA keys and such, so it definitely needs write access to them.

I'm okay with some of these changes (although that rundir change is not acceptable; it's a pain to get that right, and no one is complaining about it right now), but in general, the puppetmasterd process is entirely responsible for managing certificates.

None of these permissions change in 0.25 AFAIK; all that changes is what classes need access to the files.

You're right that if puppetmasterd is compromised then you're screwed, but that's the case with any CA, right?

If we accept these changes, then we're in the situation where creating a CA requires root access, and even worse, some small initialization step requires root and the rest of the time you should not be root.

I think this is a bad idea, so please provide better justification for the change. And again, please remove the change to the rundir; if you think there's a problem there, file a separate bug. That goes for any other bugs you're trying to fix -- each ticket is for one bug at a time.

Hi luke,

The puppetmasterd process creates the CA keys and such, so it definitely needs write access to them.

Agreed, but only as long as you stick to the "inline" CA.

Keeping your CA within a networked application and running it as the application user is the real weakness. As long as you don't separate the CA from the networked application you'll not be able to fix this issue.

I perfectly understand the trade off between usability and security considerations. I am comfortable with running my own (separate) CA, so this is not an issue for me. I didn't want to insinuate however that this should be everybody's choice!

I think the only thing you can reasonably do about the issue is:
- Disclose the risk inherent with running an "inline" CA and document the more secure version as the one recommended when your puppetmaster's port or the communication channel is exposed to some considerable threat or strong client authentication is required. Otherwise you create a false sense of security IMO.
- Provide a more secure independent CA infrastructure as an easily configurable option for those who want it. I am aware that this is already possible it's just not an especially well promoted alternative and does not have tool support... (see easyrsa that ship's with Debian's openvpn for an example). Could puppetca not be established as an optional root user level protection tool? You could probably reuse easyrsa or similar as a backend.

I'm okay with some of these changes (although that rundir change is not acceptable;

You are right. It was not my intention to mix those two up. It's a totally unrelated change that crept into the patch without me realizing it. Sorry for that.

My patch was meant as a first attempt to workaround the issue but I realized later that it didn't really fix it. :-( I have already moved on to a separate CA myself (while enforcing even more restrictive access to the PKI).

You're right that if puppetmasterd is compromised then you're screwed, but that's the case with any CA, right?

No, CAs are usually not exposed to the network. They should be run in isolation, probably on a dedicated machine. At the very least networked applications should not have write privilege on private keys, revocation lists or the directories they reside in. Applications should not have read access to the CA's root private key either.

If we accept these changes, then we're in the situation where creating a CA requires root access, and even worse, > some small initialization step requires root and the rest of the time you should not be root.

Correct, creating a CA as well as client authentication configuration and certificate signing does usually require root access.

I'm confused about exactly what you're recommending here.

Are you saying we should extract the CA into a separate application entirely and it should run as root only? If so, then I assume we have to build an API between it and puppetmasterd, so that it can still do some CA operations?

If that's not what you're recommending as an architecture, then what are you recommending?

Are you saying we should extract the CA into a separate application entirely and it should run as root only?

Yes, that's right. Either extract it to puppetca or let the user use openssl proper. I am proposing that this should be a configurable option for those users who want it. Let's say that there will be an "automatic CA setup" (=default) and a "manual CA setup".

"Automatic CA mode" is what we have today.

If the user chooses "manual CA mode", however, then puppetmasterd will no longer be able to:
- generate the the CA root certificate (above all it's private key...)
- read the CA root certificate's private key
- generate client certificate requests
- sign client certificate requests
- write to the CRL

This policy will be enforced at an OS permission level. This means that all certificates and keys as well as their containing directories will be owned by root and their permissions adjusted to least privilege. In "manual CA mode" the private root CA key should be maintained on a dedicated machine (possibly without network connection).

If so, then I assume we have to build an API between it and puppetmasterd, so that it can still do some CA operations?

The only operation puppetmasterd could continue to do is sending back signed certificates to the clients.

puppetd could continue to:
- automatically generate client certificates/keys
- send certificate requests to the central puppetmasterd machine so that they can be collected in a central place

puppetmasterd would however no longer be able to sign requests nor to generate the initial root ca certificate/key, nor would puppetd clients accept trusted root certificates sent over from puppetmasterd. These steps would have to be done manually through puppetca or easy-rsa/openssl proper.

I admit that this is far less comfortable than having puppetmasterd doing all the work. But it's the only way I see how you get better protection for your root key. People who use such an expert mode will probably be able to generate and install all certificates/keys etc. with openssl or easy-rsa, however. It only needs to be documented how puppet must be configured to set the correct permissions for keys and keep it from automatically transferring the trusted root certificate.

If you want to improve usability you might use puppetca to hide at least some of the complexity of the openssl application... Maybe something like (not very well thought out though):

puppetca --generate-root-certificate could generate the root CA certificate/key infrastructure.
puppetca --install-root-certificate could install the root certificate in the right place on the clients
puppetca --sign should continue to work normally on the dedicated CA machine.

I admit that if puppetmasterd is compromised then there are probably other possiblities of compromising the clients. But IMO this doesn't justify leaving attack vectors open where they could be blocked. Maybe there are some security experts in the puppet user community who comment on this?

jerico wrote:

I admit that if puppetmasterd is compromised then there are probably other possiblities of compromising the clients. But IMO this doesn't justify leaving attack vectors open where they could be blocked. Maybe there are some security experts in the puppet user community who comment on this?

Since that resident expert would probably be me then I should put my ten cents in. I agree attack vectors should be closed - assuming the cost-benefit is there and its a risk worth fixing rather than just accepting. The solution you propose seems reasonable too. I would comment that:

a) It doesn't take into consideration any additional controls that might reduce the attack vector and the resulting risk. Firewalls for example, not exposing the service on public networks, etc, etc.

b) It doesn't take into consideration cost/ease of use versus the gain. You are recommending that this be the "manual" option rather than the default but it is still a consideration.

c) As you've acknowledged even with the vector closed a compromise of the master effectively gives you root permissions on all the clients. I also see other easier paths to compromise clients (crudely I'd just create manifests adding new users to all the clients and reseting the root password or there are a thousand other methods - rootkits, etc, etc) rather than the CA. Hence I see exposure of the CA is a lower risk than the alternatives. So I am not sure the value is there in investing the time in a significant re-write of the CA code and closing the vector.

It sounds like what you're really saying is, we should make it so someone else can use a completely different certificate authority with Puppet. All you seem to be retaining is the ability to pass signed certs back to the client, which is pretty trivial. You say it would retain the ability to generate client certs, but you can't have that without the CA key.

When 0.25 comes out, it will be downright straightforward for you to write plugins that do all reading and writing from a central, protected host, and you can build that host however you want.

At this point, I'm willing to look at patches that explicitly point to areas where we could make it easier to integrate an external, more tightly controlled CA, but I'm not willing to modify the builtin CA to be more controlled.

You could even use Puppet's CA on that controlled machine, since it's not really root you're concerned about, it's having a networked process with write rights. Just use puppetca with no puppetmasterd.

And for the record, the current CA's entire job is to wrap openssl -- that's the whole reason it exists, so there's not much point in talking about making another wrapper for it.

You say it would retain the ability to generate client certs, but you can't have that without the CA key.

That's not correct. You certainly do not need the root CA key to generate client cert requests. You just need it to sign them.

It doesn't take into consideration any additional controls that might reduce the attack vector

We are talking about "security in depth"... Sure I have other measures in place that should avoid a remote compromise in the first place!

rootkits

A rootkit cannot enter the system without root privilege escalation. That's what we are trying to avoid by running puppetmasterd as an unprivileged user.

I'd just create manifests adding new users to all the clients

How would you do that if the process runs as an unprivileged user and all manifests (and their directories) are root protected?

You'd still gain a lot of information about the deployment and there certainly are other ways of compromising the client once you compromised puppetmasterd... You could probably serve fake manifests from memory which is effectively the same although maybe not as easy to achieve as writing a false manifest to disk if the privileges allow you to do that.

IMO it would be a very interesting discussion to find out how such memory corruption could be contained as well. But I won't start it. I have been enough of a PITA for you. ;-)

Hence I see exposure of the CA is a lower risk than the alternatives. So I am not sure the value is there in investing the time in a significant re-write of the CA code and closing the vector.

That's accepted. I understand that you have different priorities. The discussion made the topic much clearer to me after all. I think a pointer to this bug will help others as well to clarify the issue and take an appropriate decision. Maybe you can simply reference it somewhere in the documentation?

jerico wrote:

You say it would retain the ability to generate client certs, but you can't have that without the CA key.

That's not correct. You certainly do not need the root CA key to generate client cert requests. You just need it to sign them.

Sorry; your earlier post said something about puppetd generating client certs (rather than CSRs). It must have just been a typo.

I'd just create manifests adding new users to all the clients

How would you do that if the process runs as an unprivileged user and all manifests (and their directories) are root protected?

Or, preferably, owned by another non-root user.

I'm not a big fan of root being used to do much of anything.

That's accepted. I understand that you have different priorities. The discussion made the topic much clearer to me after all. I think a pointer to this bug will help others as well to clarify the issue and take an appropriate decision. Maybe you can simply reference it somewhere in the documentation?

Where would be appropriate?

Where would be appropriate?

I suggest this page: http://reductivelabs.com/trac/puppet/wiki/CertificatesAndSecurity

Would you be willing to make the edit, then, and we can just audit it?

I have inserted a sub-heading "Manual CA Configuration (optional)" on said page.

I am not sure about the "official" way of changing default permissions or whether what I documented does reliably keep the key generating mechanism from being triggered. But you'll do an audit anyway...

Looks good to me.

So then that's it. :-)