Puppet Scalability Notes
This document describes some of the issues when scaling puppet to provide service for a large number of nodes.
Please see Using Mongrel for information on how to set up Mongrel as was used in this test.
Master Performance
As of Puppet 0.22.1, the fileserver, CA methods, and configuration parsing methods all share resources within the puppetmasterd process. The consequence of this setup is that a manifest with a large number of fileserver requests will result in the fileserver methods capitalizing the resources.
Symptoms: puppetd calls to puppetmaster.getconfig or the CA methods seem to take an unreasonably long time, or timeout altogether.
Splitting off the Fileserver
Since each node will typically make a single call to get their configuration, which in turn produces a large number of file requests, it helps dramatically if the fileserver is running in a discrete process from the configuration server.
If you’re using definitions for your remote file copies, this is relatively painless to configure. Just configure the default port to be something different.
I’ll assume you’re using something like this remotefile definition:
# JJM My preferred recipe to copy files from the server.
define remotefile($source = false,
$fileserver = false,
$port = false,
$basedir = "dist",
$owner = 0, $group = 0, $mode = 640,
$recurse = true,
$ignore = ".svn",
$backup = false) {
# Some creative use of selectors to allow overrides and defaults.
$source_real = $source ? { false => $name, default => $source }
$fileserver_real = $fileserver ? { false => "puppet", default => $fileserver }
case $port {
false: { $source_uri = "puppet://$fileserver_real/$basedir/$source_real" }
default: { $source_uri = "puppet://$fileserver_real:$port/$basedir/$source_real" }
}
file { $name:
source => "$source_uri",
ignore => $ignore,
mode => $mode,
owner => $owner,
group => $group,
recurse => $recurse,
backup => $backup
}
}
So long as we use this definition whenever we copy a file from the puppetmaster, we can easily reconfigure the system to copy files from a discrete puppetmaster process running on a different port:
Remotefile { port => 8145, fileserver => "puppet1.math.ohio-state.edu" }
We also need to start a second puppetmasterd on the port:
/usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp
--logdest=/var/log/puppet/puppetmaster.8145.log --masterport=8145
With this configuration, remotefile objects will use port 8145 by default, greatly reducing load on the “main” puppetmaster process listening on 8140.
Centralised Puppet Infrastructure
<WARNING>
This does not currently work for 0.25.x nor 2.6.x (see Issue #3143 and Issue #3770 and related issues in Issue #3640 for progress on resolution). Until the issues are resolved, it is suggested to use one machine as the CA, if you can accept the fact that its a single point of failure for creating new certificates.
</WARNING>
As we are deploying puppet infrastructure in multiple sites it was important for us to have a centralized management for puppet, but a non centralized “service”, we wanted to have the option to switch between puppet masters, and did not like the idea of a single CA for all of our infrastructure. The main reason against a common ca was, as we are really spread over the world and its common to install 50+ servers in a time frame of a few hours, we didn’t want to introduce any type of dependencies.
Additionally, revoking works better this way, and well… we just wanted to make it work.
Using our solution, you could also use real root CA, your company root or self sign certificate, in some cases it could make sense not to use a self sign if you want to reuse the certificates for Apache, ldap etc.
Since all of our puppet masters are managed as well, we have one root puppet master (i.e. puppet master of the puppet masters), we called it the puppeteer. The puppeteer installation is like a regular puppet master installation.
Please note that webrick is at this time (0.24.4) unable to handle the certs in a correct way to get this setup working. As such, you will need to use something else to handle your SSL connections such as Apache. Also, you must be using a fairly recently version of puppet for the client to support it (0.24.4 works good).
We are using Apache + Mongrel: on all puppet masters you should have something like that in your Apache configuration (that’s just the ssl part):
<VirtualHost *:8140>
SSLEngine on
SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
SSLCertificateFile /var/lib/puppet/ssl/certs/your.fqdn.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/your.fqdn.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
<Location />
Then, let your second puppet master (the second level of the certificate chain) request a certificate from the puppeteer. Setup an openssl.cnf file (just store it somewhere) with the following content (adjust for your needs):
HOME = .
RANDFILE = $ENV::HOME/.rnd
[ ca ]
default_ca = CA_default
[ CA_default ]
dir = /var/lib/puppet/ssl
new_certs_dir = $dir/ca/signed
crl_dir = $dir/ca
database = $dir/index
certificate = $dir/ca/ca_crt.pem
serial = $dir/ca/serial
crl = $dir/ca/ca_crl.pem
private_key = $dir/ca/ca_key.pem
RANDFILE = $dir/private/.rand
x509_extensions = usr_cert
unique_subject = no
name_opt = ca_default
cert_opt = ca_default
default_crl_days= 30
default_days = 3650
default_md = sha1
preserve = no
policy = policy_anything
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ req ]
default_bits = 2048
default_keyfile = ./ca/ca_key.pem
default_md = sha1
prompt = no
distinguished_name = root_ca_distinguished_name
x509_extensions = v3_ca
string_mask = nombstr
[ root_ca_distinguished_name ]
commonName = XXXXXXXX
[ usr_cert ]
basicConstraints=CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer:always
nsCaRevocationUrl = https://puppeteer.your.domain.com/ca_crl.pem
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:true
keyUsage = keyCertSign, cRLSign
[ crl_ext ]
authorityKeyIdentifier=keyid:always,issuer:always
On the puppet master then copy this file to your new puppet master (e.g. /tmp):
puppetmaster=fqdn
/usr/bin/perl -p -i -e "s/XXXXXXXX/$puppetmaster/" /tmp/openssl.cnf
/usr/bin/openssl req -new -nodes -key /var/lib/puppet/ssl/ca/ca_key.pem -config /tmp/openssl.cnf -out /tmp/${puppetmaster}.csr -passin file:/var/lib/puppet/ssl/ca/private/ca.pass
Copy the ${puppetmaster}:/tmp/${puppetmaster}.csr back to the puppeteer. On the puppeteer:
touch /var/lib/puppet/ssl/index
# Sign this request with the puppeteer's CA keys
/usr/bin/openssl ca -config openssl.cnf -extfile openssl.cnf -extensions v3_ca -in ${puppetmaster}.csr -out ${puppetmaster}.pem -passin file:/var/lib/puppet/ssl/ca/private/ca.pass -batch
# Push the new certificate into place on the puppetmaster
scp ${puppetmaster}.pem ${puppetmaster}:/var/lib/puppet/ssl/ca/ca_crt.pem
In your installation process append the content of the puppeteer’s \~puppet/ssl/ca/ca_crt.pem to /var/lib/puppet/ssl/certs/ca.pem on the client
Now you should be able to use any puppet master that was signed this way.
