Puppet

# NOTE

This procedure is currently a work in progress and as such does not

completely work with puppet. The certificate authority setup

procedure has been well tested, but there's an issue with the

bundle of certificates used to authenticate remote connections to

the puppet master server. Work is underway to sort the final issues

out.

WIP - 25 Jan 2010 - eric

# Multiple Certificate Authorities

All client/server communication is secured using standard SSL x509

cryptographic certificates. In the default configuration of puppet,

the master server acts as a certificate authority which signs

client certificate signing requests (CSR's). The certificate of

this authority is a self-signed certificate, and all authentication

is performed using signed certificates.

Author: Jeff McCune Date: 5 March 2007

# Motivation

As a configuration grows, the need to bring multiple masters online

may emerge. Leveraging the x509 public key infrastructure, we're

able to adapt a single master puppet deployment to multiple

masters, each acting as their own certificate authority, and each

mutually trusting each other.

# Procedure

We'll create an example PKI:

I highly recommend creating all of this on encrypted storage. If

you're using a Mac, this can easily be accomplished with:

    hdiutil create CA.sparseimage -encryption -size 64M -fs "HFS+J" -volname CA

4 CA's:

-   CA0: Root Authority

-   CA1: Signing Authority

-   CA2: Puppet Master 1 (host specific CA)

-   CA3: Puppet Master 2 (host specific CA)

Discussion: The Root CA is long-lived and rarely used. It should

only be used to sign the certificate of subsidiary CA's within the

organization. It's private key should be locked tightly away in a

vault and never used, except to revoke compromised CA private

keys.

The Signing CA should also only be used to sign off on other CA

certificates. This allows us to use it whenever needed, and revoke

it using Root CA if it's ever lost.

The Puppet Master CA's are both signed by Signing CA1. They will

sign requests from clients running puppetd.

In one configuration, servers could check a ca-bundle which

includes only the chain leading to the certificate which signs

client certificates (CA2 in the list above). This ensures only

clients with valid certificates from a specific CA are able to

connect to the service. Likewise, clients will only trust the chain

leading to the certificate used to sign the server's SSL

certificate (CA3), which ensures clients will only trust servers

and not other clients.

Alternatively, CA2 and CA3 can be load-balanced puppetmasters which

can each verify clients issued by the other server. It's important

to keep distinct certificates per issuer so you don't get duplicate

(issuer,serial number) tuples which would prevent revocation from

working cleanly.

Setup: Environment variables are our friends:

    SERVER1="mw430mb"

    SERVER2="manage"

    DOMAIN="math.ohio-state.edu"

Make directories for each CA:

    mkdir 01_RootCA 02_SigningCA 10_${SERVER1}.${DOMAIN}_CA 11_${SERVER2}.${DOMAIN}_CA

    curl -O http://sial.org/howto/openssl/ca/Makefile

    curl -O http://sial.org/howto/openssl/ca/openssl.cnf

    cp openssl.cnf openssl.cnf.orig

Now, edit openssl.cnf to include valid subject information for each

CA.

Here's how I set things in openssl.cnf:

    [ root_ca_distinguished_name ]

    commonName = OSU Math Root Authority

    countryName = US

    stateOrProvinceName = Ohio

    localityName = Columbus

    0.organizationName = The Ohio State University

    organizationalUnitName = Department of Mathematics

In order to match the certificate extensions Puppet creates, you

also need to set the following attributes in the [v3\_ca] section.

These settings make the CA:TRUE bits critical extensions, and flag

the certificate as only usable for signing other certificates and

revocation lists. You'll want to set these extensions for all of

the CA's we create.

Just replace sial's v3\_ca section with this section:

    [ v3_ca ]

    ## CA Extensions

    # PKIX recommendations.

    subjectKeyIdentifier=hash

    authorityKeyIdentifier=keyid,issuer:always

    basicConstraints = critical,CA:true

    keyUsage = keyCertSign, cRLSign

Copy the initial configuration file to each of the 4 CA

directories:

    cp openssl.cnf Makefile 01_*CA*/

    cp openssl.cnf Makefile 02_*CA*/

    cp openssl.cnf Makefile 10_${SERVER1}.${DOMAIN}_CA/

    cp openssl.cnf Makefile 11_${SERVER2}.${DOMAIN}_CA/

I then leave everything except the commonName and nsCaRevocationUrl

field (in the v3\_req section) the same, and edit the other 3 to

descriptive CN fields:

    commonName = OSU Math Root Authority

    commonName = OSU Math Signing Authority

    commonName = server1.math.ohio-state.edu

    commonName = server2.math.ohio-state.edu

For modern OpenSSL binaries (0.9.8a or higher) you'll need to

modify the '-newkey' parameter in the Makefile as its comment

describes -- specify '-newkey rsa:2048' to explicitly set a bit

length

Also, before you create the root ca, you might want to create a

certificate that lives longer than 5 years. Edit the Makefile for

the Root CA and change the openssl command in the init section to

9125 days or what not.

Then, run "make init" in each of the 4 directories. This creates

\*self signed\* CA certificates. We need to have each CA sign off

on the one below it next.

If you look at the RootCA certificate, it's self signed, and this

is good. Note **CA:TRUE** under Basic Constraints and the Issuer is

itself.

    openssl x509 -in ca-cert.pem -text -noout

## Tier 2 CA (The Day to Day Signing CA)

We'll sign the Signing CA's certificate with the Root CA's private

key, replacing the self-signed certificate initially created with

make init.

We need to create a new Certificate Signing Request (CSR), and

create a cert from the CSR using the Root CA's private key. Here's

how:

    # Generate the new CSR

    cd 02_*CA*/

    openssl req -new -nodes -key private/ca-key.pem -config openssl.cnf -out ca.csr

    # Move the CSR from the Sub CA to the Root CA folder

    mv ca.csr ../01_*CA*

    # Sign the CSR

    cd ../01_*CA*/

    openssl ca -config openssl.cnf -extfile openssl.cnf -extensions v3_ca -in ca.csr -out ca.cert

    rm -f ca.csr

    mv ca.cert ../02_SubCA1/ca-cert.pem

Note the v3\_ca extension listed on the command line. This is very

important and what tells the Root CA to treat this request as one

from another CA rather than a request for a SSL client or server.

The key difference is that a CA certificate is used to identify

other things, while a SSL certificate use used solely to identify

the owner of it's private key.

Now, if you notice at the top of 02\_SubCA1/ca-cert.pem, the Issuer

is RootCA. We now have a certificate chain.

Ensure X509v3 **Basic Constraints: CA:TRUE** is set in the

ca-cert.pem file, if it's not, you just have a standard SSL

certificate and not a Certificate Authority certificate. SSL

certificates cannot be trusted as a certification authority and

will not work as expected. If you don't have CA:TRUE, make sure

you're using -extensions v3\_ca when signing the certificate.

Also ensure the v3 extensions

**Subject Key Identifier and Authority Key Identifier** are set

properly. These aid in the construction of trust chains, which is

exactly what we're setting up.

## Tier 3 CA's

Next, we'll repeat this process with the Client and Server CA,

using the Signing CA to sign their CSR's, rather than use the Root

CA. The Root CA directory should now be moved to a USB stick and

removed or better yet, printed out and locked in a vault. The PKI

philosophy is that the Root CA's private key shouldn't ever be used

again, unless a revocation of the Signing certificate is needed.

Create our bundle by concatenating the ca-cert.pem files into a

ca-bundle.crt file.

    cat {01_*CA*,02_*CA*,10_*CA*,11_*CA*}/ca-cert.pem >> ca-bundle.pem

If you have an existing self signed certificate, you may want to

add it to the bundle of trusted authorities now. For example, with

puppet:

    openssl x509 -in /var/lib/puppet/ca/ca_crt.pem -text >> ca-bundle.pem

# Host SSL certificates

Phew, now we've got all of our CA's setup. But we still don't have

anything suitable for encrypting network traffic, we only have keys

suitable for certifying other keys.

I make another folder specifically for host SSL data. This means

we'll use a separate private key and certificate for SSL use than

the data we already have for the CA.

    mkdir 20_ssl_host_certs

    cd 20_ssl_host_certs

    HOST="foo.math.ohio-state.edu"

    openssl req -new -nodes -newkey rsa:2048 -keyout "$HOST".key.pem -config openssl.cnf -out "$HOST".csr

I create an openssl.cnf file to keep in the 20\_ssl\_host\_certs

directory. It has everything filled out except the CN field, which

gets pulled from the HOST environment variable and looks like so:

    HOME = .

    RANDFILE = $ENV::HOME/.rnd

    [ req ]

    default_bits = 2048

    default_keyfile = private_key.pem

    distinguished_name = req_distinguished_name

    string_mask = nombstr

    req_extensions = v3_req # The extensions to add to a certificate request

    [ req_distinguished_name ]

    countryName = Country Name (2 letter code)

    countryName_default = US

    countryName_min = 2

    countryName_max = 2

    stateOrProvinceName = State or Province Name (full name)

    stateOrProvinceName_default = Ohio

    localityName = Locality Name (eg, city)

    localityName_default = Columbus

    0.organizationName = Organization Name (eg, company)

    0.organizationName_default = The Ohio State University

    organizationalUnitName = Unit Name (e.g. Department)

    organizationalUnitName_default = Department of Mathematics

    commonName = Common Name (e.g. the FQDN of the host)

    commonName_default = $ENV::HOST

    commonName_max = 64

    [ v3_req ]

    # Extensions to add to a certificate request

    basicConstraints = CA:FALSE

    keyUsage = nonRepudiation, digitalSignature, keyEncipherment

Take the CSR generated from the private key we just created and

move it into the CA directory:

    # For example;

    mv manage.math.ohio-state.edu.csr ../11_manage.math.ohio-state.edu_CA/

    cd ../11_manage.math.ohio-state.edu_CA/

Finally, sign the host SSL certificate with the CA's private key.

We're using a Tier-3 (host specific CA) for this purpose

    cd ../11_manage.math.ohio-state.edu_CA/

    make sign

Then, move the certificate back into the host SSL directory.

    mv manage.math.ohio-state.edu.cert ../20_ssl_host_certs/

You now have a private key and certificate signed by a trusted

authority suitable for SSL client/server use.

Here's how we can check the certificates:

    # First host's SSL cert

    openssl verify -verbose -purpose sslclient -CAfile ../ca-trusted-certs.pem mw430mb.math.ohio-state.edu.cert

    # Second host's SSL cert

    openssl verify -verbose -purpose sslclient -CAfile ../ca-trusted-certs.pem manage.math.ohio-state.edu.cert

## Useful OpenSSL commands

To extract a PEM encoded public key from a certificate:

    openssl x509 -pubkey -outform pem -noout < ca-cert.pem > ca-pub.pem

# Configure the Puppet Master

You'll need to configure the SSL server piece of the Puppet Master

to trust certificates signed by the authorities we just created.

    [certificates]

      localcacert = $ssldir/trusted-ca-bundle.pem

      hostprivkey = $ssldir/host_key.pem

      hostcert = $ssldir/host_cert.pem

      hostpubkey = $publickeydir/host_pub.pem

Note that trusted-ca-bundle.pem is the ca-bundle.pem file we

created above. This is also known as the Root Certificate Store, or

Trust Root, etc...

Next, we'll need to configure the Certificate Authority piece of

the Puppet Master to use the CA object we've created.

    [ca]

      cacert = $cadir/ca-cert.pem

      cakey = $cadir/ca-key.pem

      capub = $cadir/ca-pub.pem

That should do the trick.

# Troubleshooting

If you run into issues setting up this chain of trust, the

following commands will greatly help you.

To ensure you have your certificate correctly setup, eliminate

Puppet from the equation entirely. We're able to setup a SSL client

and server using only the openssl command:

On the master host:

    openssl s_server -Verify 10 -cert host_cert.pem -key host_key.pem -CAfile ca-bundle.pem

On the client host:

    cd /var/lib/puppet

    openssl s_client -connect puppet:4433 -verify 10 \

      -CAfile ssl/certs/ca.pem \

      -cert ssl/certs/$(hostname).pem \

      -key /Support/etc/puppet/ssl/private_keys/client_priv.pem

You may need to change CAfile ssl/certs/ca.pem to the path of your

ca-bundle.pem.

If you send anything via standard input to the client, it should

display on standard output of the server.