Puppet

# Using Stored Configurations

Puppet uses a technique called stored configuration, enabled with

the storeconfigs configuration option, to store data in a database.

This technique makes use of the Ruby on Rails framework and one of

three database back-ends: SQLite (the default; not suitable for production use), MySQL or

PostgreSQL. Additionally, a message queue can be used to improve a

puppetmaster's client response time. Currently stored

configurations are principally used by exported resources.

It is important to note that once a node is in the stored

configuration database it will not be automatically purged - even

if you remove the corresponding node from your configuration. If

you wish to purge nodes from your database you will need to do this

manually. There is a script in the ext directory that purges nodes

from stored configuration.

# Installing Prerequisites

To make use of stored configuration you will need to install some

prerequisites principally the database back-end you wish to use and

the Rails framework.

## Rails

**Puppet 0.24.8 and older**

Rails version 2.2.2 or earlier.

**Puppet 0.25 and later**

Rails version 2.2.2 or greater.

**Warning**

Jeff McCune <jeff@puppetlabs.com>

2010-10-01

With Puppet version 0.25.5 and MySQL on Enterprise Linux 5 you must update the Active Record gem to version 2.3.5 in order to avoid the error:

    err: Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::

    Error: MySQL server has gone away: SELECT * FROM `hosts` WHERE (`hosts`.`name` =

     'foo.puppetlabs.lan')  LIMIT 1

    warning: Not using cache on failed catalog

    err: Could not retrieve catalog; skipping run

In order to update the gem to this specific version, it is possible to install the gem into the puppet home directory:

    sudo -H -E -u puppet -s

    bash-3.2$ gem install activerecord -v 2.3.5

    WARNING:  Installing to ~/.gem since /usr/lib/ruby/gems/1.8 and

              /usr/bin aren't both writable.

    WARNING:  You don't have /var/lib/puppet/.gem/ruby/1.8/bin in your PATH,

              gem executables will not run.

    Successfully installed activesupport-2.3.5

    Successfully installed activerecord-2.3.5

    2 gems installed

    Installing ri documentation for activesupport-2.3.5...

    Installing ri documentation for activerecord-2.3.5...

    Installing RDoc documentation for activesupport-2.3.5...

    Installing RDoc documentation for activerecord-2.3.5...

    bash-3.2$ gem list

    *** LOCAL GEMS ***

    actionmailer (2.2.2)

    actionpack (2.2.2)

    activerecord (2.3.5, 2.2.3, 2.2.2)

    activeresource (2.2.2)

    activesupport (2.3.5, 2.2.3, 2.2.2)

    fastthread (1.0.7)

    passenger (2.2.11)

    rack (1.0.1)

    rails (2.2.2)

    rake (0.8.7)

    sqlite3-ruby (1.2.4)

Reference: http://projects.puppetlabs.com/issues/4694

**Installation**

We start by installing the rails Ruby Gem, either with the

distribution's package management (e.g. Debian/Ubuntu have a

'rails' package) or as a ruby gem.

To ensure a specific version of rails (not the latest available

from RubyForge) specify the optional '-v' parameter with the

version number you wish to install:

    gem install rails -v 2.2.2

## MySQL

For MySQL you will need to install a number of packages. For Red

Hat/Fedora the typical packages required would be:

    mysql

    mysql-devel

    mysql-server

Or for Debian/Ubuntu you would need the following packages.

libmysql-ruby provides the mysql gem, so installing via gems is

unneccesary:

    mysql-server

    libmysql-ruby

You may also need to install the mysql Ruby Gem (not required on

Debian/Ubuntu). You may need to specify the --with-mysql-config

option to tell Gems where to find your MySQL libraries and

headers:

    # gem install mysql -- --with-mysql-config=/usr/bin/mysql_config

You may require the ruby-devel package in order to install this

gem.

A word of warning - it is possible to be using stored

configurations with MySQL without installing the mysql gem. This is

because rails includes with it a simple Ruby based MySQL connector.

Relying on this connector is strongly discouraged, as it has been

observed to leak filehandles, leave open network and database

connections, and eventually cause puppetmasterd to become

completely unresponsive.

In order to check if you are using the unreliable Ruby based

connector, examine the rails.log file produced by puppetmasterd. If

you are using the Ruby one, you will find a message like this:

    WARNING: You're using the Ruby-based MySQL library that ships with Rails.

    This library is not suited for production. Please install the C-based MySQL

    library instead (gem install mysql).

If you find that warning, make sure you install the mysql gem,

which is based on the native MySQL libraries, before you begin

using stored configurations in production.

## SQLite3

For SQLite you will need to install the database package. For Red

Hat/Fedora the package required would be:

    sqlite

For Ubuntu/Debian, the packages required are (libsqlite3-ruby

contains the appropriate gem):

    sqlite3

    libsqlite3-ruby

You may also need to install the sqlite3-ruby Ruby Gem (not

required on Debian/Ubuntu):

    # gem install sqlite3-ruby

## PostgreSQL

For PostgreSQL you will need to install a number of packages. For

Red Hat/Fedora the typical packages required would be:

    postgresql

    postgresql-server

    postgresql-devel

You will also need to install the postgres Ruby Gem:

    # gem install postgres

or on more recent install:

    # gem install pg

If you fail to install the postgres gem then you will receive the

following error:

    err: Could not retrieve configuration: Uncaught exception no such file to load -- postgres in method puppetmaster.getconfig

If you get the error:

    err: Could not retrieve catalog from remote server: Error 400 on SERVER: NoMethodError: undefined method `fformat' for #<PGresult:0x.........>: SHOW client_min_messages

Try the pg gem instead of the postgres one.

For PostgreSQL you will also need to create the database and create

an appropriate role and password.

# Configuring basic storeconfigs

Now we need to configure stored configuration in Puppet itself.

There are three types of database adapters available for us, the

default sqlite3, MySQL and PostgreSQL. For the latter back-end

databases we also have available a number of options that let us

specify a particular user, password and the location of a database

socket.

Let's begin by walking through the steps to configure the SQLite backend.

**Note that this backend is not suitable for production-level use** -- it

can't handle high volume traffic, and will report locking failures if used

with more than five or ten hosts.

-   Configure puppet.conf:

        [puppetmasterd]

        storeconfigs = true

        dbadapter = sqlite3

-   "Thin" stored configuration (optional) - from 0.25.0 we can

    also use the "thin" stored configuration option. This only collects

    and stores to the database exported resources, tags and host facts.

    To use "thin" stored configuration configure it in your

    puppet.conf:

        [puppetmasterd]

        storeconfigs = true

        thin_storeconfigs = true

        dbadapter = sqlite3

-   Start the puppetmasterd daemon

-   Start the puppetd client

Now storeconfigs should be enabled and running.

Note that if you intend to use PuppetShow, you may need to include

a dblocation setting under puppetmasterd, pointing to the default

location.

# Configuring storeconfigs for MySQL

-   Create a puppet database, and grant privileges. Replace

    password below with an appropriate password:

        # mysql -u root -p

        mysql> create database puppet;

        mysql> grant all privileges on puppet.* to puppet@localhost identified by 'password';

-   Configure your [puppetmasterd] section to reflect these

    settings:

        [puppetmasterd]

        storeconfigs = true

        dbadapter = mysql

        dbuser = puppet

        dbpassword = password

        dbserver = localhost

        dbsocket = /var/run/mysqld/mysqld.sock

-   Start the puppetmasterd daemon

-   Start the puppetd client

You can also find some information on

[[My\_Sql\_Stored\_Configuration\_Patterns|Recipes/MySQLStored

Configuration]] .

To optimize some often run Puppet queries on your MySQL database,

use the following index:

    create index exported_restype_title on resources (exported, restype, title(50));

# Configuring storeconfigs for PostgreSQL

By default (on RHEL/CentOS, other distros unknown), you will have

to su - to the postgres user to do any administrative work such as

database creation.

-   Set up the database:

        # su - postgres

        $ psql template1

        template1=# create database puppet;

        CREATE DATABASE

        create user puppet with unencrypted password 'password';

        CREATE ROLE

        template1=# grant create on database puppet to puppet;

Note that Puppet requires the CREATE right to do automatic table

creation to initialize the database.

-   Configure your [puppetmasterd] section to reflect these

    settings:

        [puppetmasterd]

        storeconfigs = true

        dbadapter = postgresql

        dbuser = puppet

        dbpassword = password

        dbserver = localhost

        dbname = puppet

-   Start the puppetmasterd daemon

-   Start the puppetd client

-   Check /var/log/messages (or wherever you have Puppet set to

    log) for the message for an ''Initialized database'' message when

    the first client checks in.

# Installing Queuing Support for Storeconfigs

Puppet queueing is a feature which is designed to take some load

off of the Puppet master by transferring the task of updating the

database to a separate program which is named puppetqd (the Puppet

Queue Daemon).

To add support for queuing to an existing storeconfigs

configuration, you only need to install a queue broker, configure

your Puppet master, and configure your puppetqd.

While many message queue services exist, currently Puppet only

supports the Stomp protocol. We have tested using StompServer and

ActiveMQ. While StompServer is easy to set up and configure, we

suggest you use ActiveMQ -- if you need queueing, you need it

because of scalability, and StompServer simply isn't built for

scaling up to enterprise needs.

Queuing is available from release 0.25.0 onwards. It is NOT

available in the 0.24.x releases.

## Installing StompServer

First make sure you have the Ruby development package. On a Red

Hat-based system, this would be:

    yum install ruby-devel

Or on a Debian or Ubuntu-based system:

    apt-get install ruby1.8-dev

StompServer is available as a gem, so you can install via:

    gem install stompserver

## Installing ActiveMQ

Download and install ActiveMQ: e.g.,:

    $ wget http://www.apache.org/dyn/closer.cgi/activemq/apache-activemq/5.2.0/apache-activemq-5.2.0-bin.tar.gz

    $ tar zxvf apache-activemq-5.2.0-bin.tar.gz

# Configuring queuing storeconfigs

## Configuring the queuing broker

### Configuring StompServer

Configuration can either be done on the command line or via a

config file. For example, the following command-line:

    stompserver -q dbm

will start the stompserver on the default port of 61613 and use a

DBM file as its backing store (i.e., while messages have not yet

been picked up by puppetqd, they will be stored in a DBM file).

The configuration file:

    $ cat /etc/puppet/puppet-queue.conf

    :daemon: true

    :working_dir: /tmp/stompserver

    :storage: .queue

    :queue: file

    :auth: false

    :debug: false

    :group:

    :user:

    :host: 127.0.0.1

    :port: 61613

configures the stompserver to use a file backing store.

Then run the stompserver:

    stompserver -C /etc/puppet/puppet-queue.conf

Consult the stompserver's README.txt for additional information on

installation and configuration.

### Configuring ActiveMQ

Create an activemq.xml file (based on the samples provided in the

source), and enable stomp. The sample configuration uses the

default backing store, AMQ, (which is a transactional journal in

ActiveMQ 5) and only requires the one-line change below to enable

stomp on port 61613:

    <transportConnectors> <transportConnector name="stomp" uri="stomp://localhost:61613"/> </transportConnectors>

You can then start ActiveMQ:

    activemq

You should consult the ActiveMQ documentation for additional

information on configuring ActiveMQ.

## Configuring Puppet

You need to configure both puppetmasterd and puppetqd to use

queuing.

In this example, we're keeping our stored configuration information

in sqlite3 -- if you're using PostgreSQL, MySQL, or others, your

dbadapter and dblocation information will differ).

Please refer to the basic stored configuration section on the wiki

for details about creating the database for MySQL or PostgreSQL.

First we need to install the Stomp client. You can do this from a

gem:

    gem install stomp

Next, we need to configure Puppet itself. Note that the

configuration of the Puppet master and the puppetqd is the same:

this is because they both need to read information out of the same

database (i.e., the database is a cache). Thus, a puppet.conf would

contain:

    [main]

      queue_type = stomp

      queue_source = stomp://localhost:61613

      dbadapter = sqlite3

    [puppetmasterd]

      async_storeconfigs = true

Note that these can all be put onto a command line:

    puppetmasterd --async_storeconfigs --queue_type=stomp --queue_source="stomp://localhost:61613" \

      --dbadapter sqlite3

You then need to run the puppetqd daemon.

> puppetqd

# Bootstrapping into stored configuration

If you have a lot of hosts, turning on stored configuration might

crush your database server. The main problem is that data for every

host needs to be INSERTed into the database. Once you get past that

hurdle there's much less SQL activity. Here are some notes about

how we (usg.edu folks) got over the hurdle. The main bottleneck for

us was CPU--IO wait wasn't an issue.

First we tuned our MySQL server (all based on masterzen's

recommendations). We added these lines to the [mysqld] section of

my.cnf:

    innodb_buffer_pool_size=2G

    innodb_log_file_size=256M

    innodb_log_buffer_size=64M

    innodb_additional_mem_pool_size=20M

    innodb_flush_method = O_DIRECT

Next we stopped puppetd on half of our ~400 servers. This, plus

the MySQL changes above, allowed our server to survive the initial

stored configuration hit of our other 200 servers. After the first

200 were in stored configuration we ran puppetd --test --tags

SMALLMODULE on the second half of our hosts. Later we restarted

puppetd.

Our clients check in once an hour. About 15 hosts check in per

minute. The puppetmasterd and mysqld run on the same server (along

with a few other applications). Our server has 8 CPUs and 12G of

RAM.

Here's a fine blog post by masterzen about this issue:

[http://www.masterzen.fr/2009/03/18/omg-storedconfigs-killed-my-database/](http://www.masterzen.fr/2009/03/18/omg-storedconfigs-killed-my-database/)

# Cleaning Stored Configs

On occasion, something about a system changes and you will need to clean out the stored configuration database for a given node.  This is fairly harmless to do because the information will repopulate the next time the node checks in with the master.

Download the cleaner script contained with the source:

https://github.com/puppetlabs/puppet/raw/master/ext/puppetstoredconfigclean.rb

Once you have the script in your puppet master, you can clean the information out of your stored configuration for a given node, but running the script like so:

    ./puppetstoredconfigclean.rb host1.example.com

This will remove all of the information about the node 'host1.example.com' from your stored configs database.

For more information on usage, you can run the script with --help like so:

    ./puppetstoredconfigclean.rb --help