Puppet

# Puppet Scalability Notes

This document describes some of the issues when scaling puppet to

provide service for a large number of nodes.

Please see [[Using Mongrel]] for information on how to set up

Mongrel as was used in this test.

# Master Performance

As of Puppet 0.22.1, the fileserver, CA methods, and configuration

parsing methods all share resources within the puppetmasterd

process. The consequence of this setup is that a manifest with a

large number of fileserver requests will result in the fileserver

methods capitalizing the resources.

Symptoms: puppetd calls to puppetmaster.getconfig or the CA methods

seem to take an unreasonably long time, or timeout altogether.

# Splitting off the Fileserver

Since each node will typically make a single call to get their

configuration, which in turn produces a large number of file

requests, it helps dramatically if the fileserver is running in a

discrete process from the configuration server.

If you're using definitions for your remote file copies, this is

relatively painless to configure. Just configure the default port

to be something different.

I'll assume you're using something like this remotefile

definition:

    # JJM My preferred recipe to copy files from the server.

    define remotefile($source = false,

      $fileserver = false,

      $port = false,

      $basedir = "dist",

      $owner = 0, $group = 0, $mode = 640,

      $recurse = true,

      $ignore = ".svn",

      $backup = false) {

        # Some creative use of selectors to allow overrides and defaults.

        $source_real = $source ? { false => $name, default => $source }

        $fileserver_real = $fileserver ? { false => "puppet", default => $fileserver }

        case $port {

                false: { $source_uri = "puppet://$fileserver_real/$basedir/$source_real" }

                default: { $source_uri = "puppet://$fileserver_real:$port/$basedir/$source_real" }

        }

        file { $name:

                source => "$source_uri",

                ignore => $ignore,

                mode => $mode,

                owner => $owner,

                group => $group,

                recurse => $recurse,

                backup => $backup

        }

    }

So long as we use this definition whenever we copy a file from the

puppetmaster, we can easily reconfigure the system to copy files

from a discrete puppetmaster process running on a different port:

    Remotefile { port => 8145, fileserver => "puppet1.math.ohio-state.edu" }

We also need to start a second puppetmasterd on the port:

    /usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp

    --logdest=/var/log/puppet/puppetmaster.8145.log --masterport=8145

With this configuration, remotefile objects will use port 8145 by

default, greatly reducing load on the "main" puppetmaster process

listening on 8140.

# Centralised Puppet Infrastructure

**\<WARNING\>** 

This does not currently work for 0.25.x nor 2.6.x (see Issue #3143 

and Issue #3770 and related issues in Issue #3640 for progress on resolution). 

Until the issues are resolved, it is suggested to use one machine as the CA, 

if you can accept the fact that its a single point of failure for creating new certificates.

**\</WARNING\>** 

As we are deploying puppet infrastructure in multiple sites it was

important for us to have a centralized management for puppet, but a

non centralized "service", we wanted to have the option to switch

between puppet masters, and did not like the idea of a single CA

for all of our infrastructure. The main reason against a common ca

was, as we are really spread over the world and its common to

install 50+ servers in a time frame of a few hours, we didn't want

to introduce any type of dependencies.

Additionally, revoking works better this way, and well... we just

wanted to make it work.

Using our solution, you could also use real root CA, your company

root or self sign certificate, in some cases it could make sense

not to use a self sign if you want to reuse the certificates for

Apache, ldap etc.

Since all of our puppet masters are managed as well, we have one

root puppet master (i.e. puppet master of the puppet masters), we

called it the puppeteer. The puppeteer installation is like a

regular puppet master installation.

Please note that webrick is at this time (0.24.4) unable to handle

the certs in a correct way to get this setup working. As such, you

will need to use something else to handle your SSL connections such

as Apache. Also, you must be using a fairly recently version of

puppet for the client to support it (0.24.4 works good).

We are using Apache + Mongrel: on all puppet masters you should

have something like that in your Apache configuration (that's just

the ssl part):

    <VirtualHost *:8140>

        SSLEngine on

        SSLCipherSuite          SSLv2:-LOW:-EXPORT:RC4+RSA

        SSLCertificateFile      /var/lib/puppet/ssl/certs/your.fqdn.com.pem

        SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/your.fqdn.com.pem

        SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem

        SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem

        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem

        SSLVerifyClient         optional

        SSLVerifyDepth          3

        SSLOptions              +StdEnvVars

        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e

        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e

        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    <Location />

Then, let your second puppet master (the second level of the

certificate chain) request a certificate from the puppeteer. Setup

an openssl.cnf file (just store it somewhere) with the following

content (adjust for your needs):

    HOME                    = .

    RANDFILE                = $ENV::HOME/.rnd

    [ ca ]

    default_ca      = CA_default

    [ CA_default ]

    dir             = /var/lib/puppet/ssl

    new_certs_dir   = $dir/ca/signed

    crl_dir         = $dir/ca

    database        = $dir/index

    certificate     = $dir/ca/ca_crt.pem

    serial          = $dir/ca/serial

    crl             = $dir/ca/ca_crl.pem

    private_key     = $dir/ca/ca_key.pem

    RANDFILE        = $dir/private/.rand

    x509_extensions = usr_cert

    unique_subject  = no

    name_opt        = ca_default

    cert_opt        = ca_default

    default_crl_days= 30

    default_days    = 3650

    default_md      = sha1

    preserve        = no

    policy          = policy_anything

    [ policy_match ]

    countryName             = match

    stateOrProvinceName     = match

    organizationName        = match

    organizationalUnitName  = optional

    commonName              = supplied

    emailAddress            = optional

    [ policy_anything ]

    countryName             = optional

    stateOrProvinceName     = optional

    localityName            = optional

    organizationName        = optional

    organizationalUnitName  = optional

    commonName              = supplied

    emailAddress            = optional

    [ req ]

    default_bits            = 2048

    default_keyfile         = ./ca/ca_key.pem

    default_md              = sha1

    prompt                  = no

    distinguished_name      = root_ca_distinguished_name

    x509_extensions = v3_ca

    string_mask = nombstr

    [ root_ca_distinguished_name ]

    commonName = XXXXXXXX

    [ usr_cert ]

    basicConstraints=CA:FALSE

    subjectKeyIdentifier=hash

    authorityKeyIdentifier=keyid,issuer:always

    nsCaRevocationUrl               = https://puppeteer.your.domain.com/ca_crl.pem

    [ v3_req ]

    basicConstraints = CA:FALSE

    keyUsage = nonRepudiation, digitalSignature, keyEncipherment

    [ v3_ca ]

    subjectKeyIdentifier=hash

    authorityKeyIdentifier=keyid,issuer:always

    basicConstraints = critical,CA:true

    keyUsage = keyCertSign, cRLSign

    [ crl_ext ]

    authorityKeyIdentifier=keyid:always,issuer:always

On the puppet master then copy this file to your new puppet master

(e.g. /tmp):

    puppetmaster=fqdn

    /usr/bin/perl -p -i -e "s/XXXXXXXX/$puppetmaster/" /tmp/openssl.cnf

    /usr/bin/openssl req -new -nodes -key /var/lib/puppet/ssl/ca/ca_key.pem -config /tmp/openssl.cnf -out /tmp/${puppetmaster}.csr -passin file:/var/lib/puppet/ssl/ca/private/ca.pass

Copy the ${puppetmaster}:/tmp/${puppetmaster}.csr back to the

puppeteer. On the puppeteer:

    touch /var/lib/puppet/ssl/index

    # Sign this request with the puppeteer's CA keys

    /usr/bin/openssl ca -config openssl.cnf -extfile openssl.cnf -extensions v3_ca -in ${puppetmaster}.csr -out ${puppetmaster}.pem -passin file:/var/lib/puppet/ssl/ca/private/ca.pass -batch

    # Push the new certificate into place on the puppetmaster

    scp ${puppetmaster}.pem ${puppetmaster}:/var/lib/puppet/ssl/ca/ca_crt.pem

In your installation process append the content of the puppeteer's

\~puppet/ssl/ca/ca\_crt.pem to /var/lib/puppet/ssl/certs/ca.pem on

the client

Now you should be able to use any puppet master that was signed

this way.