Puppet

We have two puppet-master installations where stored configs configured using MySQL.

Very often, puppet master reports:

Thu Oct 21 18:48:19 +0400 2010 //host.domain.com/Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::Error: MySQL server has gone away: SELECT * FROM `hosts` WHERE (`hosts`.`name` = 'host.domain.com')  LIMIT 1
Thu Oct 21 18:48:19 +0400 2010 //host.domain.com/Puppet (notice): Using cached catalog
Thu Oct 21 18:48:19 +0400 2010 //host.domain.com/Puppet (err): Could not retrieve catalog; skipping run

SQL may differ.

Every master controls 5-10 agents (each agent runs every 5 minutes), but we plan to control total 200+ hosts, and this behavior will be very annoying under high load.

Message reported for different agent hosts, time between reports differs from 5 to 120 minutes.

Software used on affected servers: Linux system is CentOS 5/32 (XEN VM) * Puppet 2.6.1 * Nginx * Mongrel * MySQL 5.0.77 * Rails and ActiveRecord 2.2.2 * Ruby 1.8.6-111

FreeBSD system is 7.3/amd64 * Puppet 2.6.2 * Nginx * Mongrel * MySQL 5.1.48 * Rails and ActiveRecord 2.2.2 * Ruby 1.8.7-248

All mysql servers with default configuration (on FreeBSD – buffers configured) with InnoDB enabled, and Puppet tables in InnoDB.

Puppet config from Linux (on BSD slighty differs):

[main]
    vardir = /var/lib/puppet
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    report = true

[master]
    bindaddress = 0.0.0.0
    reports = log,tagmail,store
    syslogfacility = user
    modulepath = /etc/puppet/modules/main:/etc/puppet/modules/custom:/usr/share/puppet/modules
    storeconfigs = true
    dbadapter = mysql
    dbuser = puppet
    dbpassword = password
    dbserver = localhost
    dbsocket = /var/lib/mysql/mysql.sock