History

NOTE

This procedure is currently a work in progress and as such does not completely work with puppet. The certificate authority setup procedure has been well tested, but there’s an issue with the bundle of certificates used to authenticate remote connections to the puppet master server. Work is underway to sort the final issues out.

WIP – 25 Jan 2010 – eric

Multiple Certificate Authorities

All client/server communication is secured using standard SSL x509 cryptographic certificates. In the default configuration of puppet, the master server acts as a certificate authority which signs client certificate signing requests (CSR’s). The certificate of this authority is a self-signed certificate, and all authentication is performed using signed certificates.

Author: Jeff McCune Date: 5 March 2007

Motivation

As a configuration grows, the need to bring multiple masters online may emerge. Leveraging the x509 public key infrastructure, we’re able to adapt a single master puppet deployment to multiple masters, each acting as their own certificate authority, and each mutually trusting each other.

Procedure

We’ll create an example PKI:

I highly recommend creating all of this on encrypted storage. If you’re using a Mac, this can easily be accomplished with:

hdiutil create CA.sparseimage -encryption -size 64M -fs "HFS+J" -volname CA

4 CA’s:

CA0: Root Authority
CA1: Signing Authority
CA2: Puppet Master 1 (host specific CA)
CA3: Puppet Master 2 (host specific CA)

Discussion: The Root CA is long-lived and rarely used. It should only be used to sign the certificate of subsidiary CA’s within the organization. It’s private key should be locked tightly away in a vault and never used, except to revoke compromised CA private keys.

The Signing CA should also only be used to sign off on other CA certificates. This allows us to use it whenever needed, and revoke it using Root CA if it’s ever lost.

The Puppet Master CA’s are both signed by Signing CA1. They will sign requests from clients running puppetd.

In one configuration, servers could check a ca-bundle which includes only the chain leading to the certificate which signs client certificates (CA2 in the list above). This ensures only clients with valid certificates from a specific CA are able to connect to the service. Likewise, clients will only trust the chain leading to the certificate used to sign the server’s SSL certificate (CA3), which ensures clients will only trust servers and not other clients.

Alternatively, CA2 and CA3 can be load-balanced puppetmasters which can each verify clients issued by the other server. It’s important to keep distinct certificates per issuer so you don’t get duplicate (issuer,serial number) tuples which would prevent revocation from working cleanly.

Setup: Environment variables are our friends:

SERVER1="mw430mb"
SERVER2="manage"
DOMAIN="math.ohio-state.edu"

Make directories for each CA:

mkdir 01_RootCA 02_SigningCA 10_${SERVER1}.${DOMAIN}_CA 11_${SERVER2}.${DOMAIN}_CA
curl -O http://sial.org/howto/openssl/ca/Makefile
curl -O http://sial.org/howto/openssl/ca/openssl.cnf
cp openssl.cnf openssl.cnf.orig

Now, edit openssl.cnf to include valid subject information for each CA.

Here’s how I set things in openssl.cnf:

[ root_ca_distinguished_name ]
commonName = OSU Math Root Authority
countryName = US
stateOrProvinceName = Ohio
localityName = Columbus
0.organizationName = The Ohio State University
organizationalUnitName = Department of Mathematics

In order to match the certificate extensions Puppet creates, you also need to set the following attributes in the [v3_ca] section. These settings make the CA:TRUE bits critical extensions, and flag the certificate as only usable for signing other certificates and revocation lists. You’ll want to set these extensions for all of the CA’s we create.

Just replace sial’s v3_ca section with this section:

[ v3_ca ]
## CA Extensions
# PKIX recommendations.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:true
keyUsage = keyCertSign, cRLSign

Copy the initial configuration file to each of the 4 CA directories:

cp openssl.cnf Makefile 01_*CA*/
cp openssl.cnf Makefile 02_*CA*/
cp openssl.cnf Makefile 10_${SERVER1}.${DOMAIN}_CA/
cp openssl.cnf Makefile 11_${SERVER2}.${DOMAIN}_CA/

I then leave everything except the commonName and nsCaRevocationUrl field (in the v3_req section) the same, and edit the other 3 to descriptive CN fields:

commonName = OSU Math Root Authority
commonName = OSU Math Signing Authority
commonName = server1.math.ohio-state.edu
commonName = server2.math.ohio-state.edu

For modern OpenSSL binaries (0.9.8a or higher) you’ll need to modify the ‘-newkey’ parameter in the Makefile as its comment describes — specify ‘-newkey rsa:2048’ to explicitly set a bit length

Also, before you create the root ca, you might want to create a certificate that lives longer than 5 years. Edit the Makefile for the Root CA and change the openssl command in the init section to 9125 days or what not.

Then, run “make init” in each of the 4 directories. This creates *self signed* CA certificates. We need to have each CA sign off on the one below it next.

If you look at the RootCA certificate, it’s self signed, and this is good. Note CA:TRUE under Basic Constraints and the Issuer is itself.

openssl x509 -in ca-cert.pem -text -noout

Tier 2 CA (The Day to Day Signing CA)

We’ll sign the Signing CA’s certificate with the Root CA’s private key, replacing the self-signed certificate initially created with make init.

We need to create a new Certificate Signing Request (CSR), and create a cert from the CSR using the Root CA’s private key. Here’s how:

# Generate the new CSR
cd 02_*CA*/
openssl req -new -nodes -key private/ca-key.pem -config openssl.cnf -out ca.csr

# Move the CSR from the Sub CA to the Root CA folder
mv ca.csr ../01_*CA*

# Sign the CSR
cd ../01_*CA*/
openssl ca -config openssl.cnf -extfile openssl.cnf -extensions v3_ca -in ca.csr -out ca.cert
rm -f ca.csr
mv ca.cert ../02_SubCA1/ca-cert.pem

Note the v3_ca extension listed on the command line. This is very important and what tells the Root CA to treat this request as one from another CA rather than a request for a SSL client or server. The key difference is that a CA certificate is used to identify other things, while a SSL certificate use used solely to identify the owner of it’s private key.

Now, if you notice at the top of 02_SubCA1/ca-cert.pem, the Issuer is RootCA. We now have a certificate chain.

Ensure X509v3 Basic Constraints: CA:TRUE is set in the ca-cert.pem file, if it’s not, you just have a standard SSL certificate and not a Certificate Authority certificate. SSL certificates cannot be trusted as a certification authority and will not work as expected. If you don’t have CA:TRUE, make sure you’re using -extensions v3_ca when signing the certificate.

Also ensure the v3 extensions Subject Key Identifier and Authority Key Identifier are set properly. These aid in the construction of trust chains, which is exactly what we’re setting up.

Tier 3 CA’s

Next, we’ll repeat this process with the Client and Server CA, using the Signing CA to sign their CSR’s, rather than use the Root CA. The Root CA directory should now be moved to a USB stick and removed or better yet, printed out and locked in a vault. The PKI philosophy is that the Root CA’s private key shouldn’t ever be used again, unless a revocation of the Signing certificate is needed.

Create our bundle by concatenating the ca-cert.pem files into a ca-bundle.crt file.

cat {01_*CA*,02_*CA*,10_*CA*,11_*CA*}/ca-cert.pem >> ca-bundle.pem

If you have an existing self signed certificate, you may want to add it to the bundle of trusted authorities now. For example, with puppet:

openssl x509 -in /var/lib/puppet/ca/ca_crt.pem -text >> ca-bundle.pem

Host SSL certificates

Phew, now we’ve got all of our CA’s setup. But we still don’t have anything suitable for encrypting network traffic, we only have keys suitable for certifying other keys.

I make another folder specifically for host SSL data. This means we’ll use a separate private key and certificate for SSL use than the data we already have for the CA.

mkdir 20_ssl_host_certs
cd 20_ssl_host_certs
HOST="foo.math.ohio-state.edu"
openssl req -new -nodes -newkey rsa:2048 -keyout "$HOST".key.pem -config openssl.cnf -out "$HOST".csr

I create an openssl.cnf file to keep in the 20_ssl_host_certs directory. It has everything filled out except the CN field, which gets pulled from the HOST environment variable and looks like so:

HOME = .
RANDFILE = $ENV::HOME/.rnd

[ req ]
default_bits = 2048
default_keyfile = private_key.pem
distinguished_name = req_distinguished_name
string_mask = nombstr
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = US
countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Ohio

localityName = Locality Name (eg, city)
localityName_default = Columbus

0.organizationName = Organization Name (eg, company)
0.organizationName_default = The Ohio State University
organizationalUnitName = Unit Name (e.g. Department)
organizationalUnitName_default = Department of Mathematics

commonName = Common Name (e.g. the FQDN of the host)
commonName_default = $ENV::HOST
commonName_max = 64

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

Take the CSR generated from the private key we just created and move it into the CA directory:

# For example;
mv manage.math.ohio-state.edu.csr ../11_manage.math.ohio-state.edu_CA/
cd ../11_manage.math.ohio-state.edu_CA/

Finally, sign the host SSL certificate with the CA’s private key. We’re using a Tier-3 (host specific CA) for this purpose

cd ../11_manage.math.ohio-state.edu_CA/
make sign

Then, move the certificate back into the host SSL directory.

mv manage.math.ohio-state.edu.cert ../20_ssl_host_certs/

You now have a private key and certificate signed by a trusted authority suitable for SSL client/server use.

Here’s how we can check the certificates:

# First host's SSL cert
openssl verify -verbose -purpose sslclient -CAfile ../ca-trusted-certs.pem mw430mb.math.ohio-state.edu.cert
# Second host's SSL cert
openssl verify -verbose -purpose sslclient -CAfile ../ca-trusted-certs.pem manage.math.ohio-state.edu.cert

Useful OpenSSL commands

To extract a PEM encoded public key from a certificate:

openssl x509 -pubkey -outform pem -noout < ca-cert.pem > ca-pub.pem

Configure the Puppet Master

You’ll need to configure the SSL server piece of the Puppet Master to trust certificates signed by the authorities we just created.

[certificates]
  localcacert = $ssldir/trusted-ca-bundle.pem
  hostprivkey = $ssldir/host_key.pem
  hostcert = $ssldir/host_cert.pem
  hostpubkey = $publickeydir/host_pub.pem

Note that trusted-ca-bundle.pem is the ca-bundle.pem file we created above. This is also known as the Root Certificate Store, or Trust Root, etc…

Next, we’ll need to configure the Certificate Authority piece of the Puppet Master to use the CA object we’ve created.

[ca]
  cacert = $cadir/ca-cert.pem
  cakey = $cadir/ca-key.pem
  capub = $cadir/ca-pub.pem

That should do the trick.

Troubleshooting

If you run into issues setting up this chain of trust, the following commands will greatly help you.

To ensure you have your certificate correctly setup, eliminate Puppet from the equation entirely. We’re able to setup a SSL client and server using only the openssl command:

On the master host:

openssl s_server -Verify 10 -cert host_cert.pem -key host_key.pem -CAfile ca-bundle.pem

On the client host:

cd /var/lib/puppet
openssl s_client -connect puppet:4433 -verify 10 \
  -CAfile ssl/certs/ca.pem \
  -cert ssl/certs/$(hostname).pem \
  -key /Support/etc/puppet/ssl/private_keys/client_priv.pem

You may need to change CAfile ssl/certs/ca.pem to the path of your ca-bundle.pem.

If you send anything via standard input to the client, it should display on standard output of the server.

Puppet

New To Puppet

Release Notes

References and Advanced Topics

Wiki