History

Generating a config file from fragments

Introduction

Many daemons, such as Apache or Monit support loading configuration from several independent files. However, plenty of daemons only support one configuration file, which is problematic for puppet, as each slightly different use-case would require it’s own, complete configuration file and fixes made to one file would not automatically propagate to others. There are two basic approaches to solving this problem with Puppet:

  • Add/remove/modify the lines in the configuration file
  • Compile the configuration file from fragments

Here we take the latter approach. To assemble configuration files from fragments we need a

  • Directory to store the fragments in
  • Core configuration file
  • Support script that concatenates the fragments together and triggers a service refresh. Alternatively the refresh could be done with Notify using Puppet.
  • Exec resource that triggers concatenation script

All of these are added to the core configuration class. In addition all classes that add configuration fragments need supporting code.

Concatenation script setup

First we need a script to concatenate the configuration files. Something like works despite it’s simplicity:

#!/bin/bash

# First argument ($1): directory containing the configuration file fragments
# Second argument ($2): the path to resulting file
# Third argument ($3): the service name for restarting it

# Remove results of previous runs
rm -f $2

# Concatenate the fragments
for FRAGMENT in `ls $1`; do
        cat $1/$FRAGMENT >> $2
done

# We should use "notify" mechanism in puppet instead, but this works atm.
/etc/init.d/$3 restart

Make sure the exec resource that does the refreshing depends on this file (or the class that installs it).

Setting up the fragment directory

Next we need a directory for the fragments. I’ve used the config-file-name.d notation:

    file { "$etc/postfix/main.cf.d":
            ensure => directory,
    }

Installing core configuration file

Next we install the core configuration file fragment that all instances of this service will use, here in the postfix::common::config subclass:

file { "core-main.cf":
    name => "$etc/postfix/main.cf.d/00core-main.cf",
    ensure => present,
        owner => root,
        group => $admingroup,
        mode  => 644,
    content => template("postfix/00core-main.cf.erb"),
    require => Class["postfix::install"],
}

Note how the fragment is name 00core-something; this makes sure it is loaded first, before any other fragments. For some services (e.g. firewall rules) this is essential, for others (e.g. monit rules) it’s not.

Finally, we make sure the core configuration file fragment is tracked for changes. If it is changed, the concatenation script is run and the service is restarted:

exec { "core-main.cf-refresh":
    command => "/usr/local/bin/concatenate_configs.sh $etc/postfix/main.cf.d $etc/postfix/main.cf postfix",
    cwd => "$etc/postfix/main.cf.d",
    path => ["/bin", "/usr/bin", "/usr/sbin", "/usr/local/bin"],
    refreshonly => true,
    subscribe => File["core-main.cf"],
    notify => Class["postfix::service"],
    require => Class["scripts"],
}

NOTE: Each configuration file fragment has to tracked separately in their respective class/module. This is somewhat redundant, but I have not yet found a way around it.

Installing fragments from other classes

Adding new fragments is identical to adding the core configuration. This specific fragment is installed and tracked by postfix::config::inbound subclass, but nothing is stopping you from importing fragments from other modules (namespaces) as well:

file { "inbound-main.cf":
    name => "$etc/postfix/main.cf.d/10inbound-main.cf",
    ensure => present,
    owner => root,
    group => $admingroup,
    mode  => 644,
    content => template("postfix/10inbound-main.cf.erb"),
    require => Class["postfix::config::common"],
}

Next we need to track this fragment for changes, and refresh the configuration with concatenation script if it changes:

exec { "inbound-main.cf-refresh":
    command => "/usr/local/bin/concatenate_configs.sh $etc/postfix/main.cf.d $etc/postfix/main.cf postfix",
    cwd => "$etc/postfix/main.cf.d",
    path => ["/bin", "/usr/bin", "/usr/sbin", "/usr/local/bin"],
    refreshonly => true,
    subscribe => File["inbound-main.cf"],
    require => Class["scripts"],
}

Variant: iptables/ip6tables ruleset from fragments

This example sets up iptables rules for both IPv4 and IPv6 from fragments. Services can then add their own iptables/ip6tables rule fragments easily. Unlike the postfix module above, this module uses two auxiliary scripts, iptables.sh and ip6tables.sh which take the place of the concatenation script. In addition, binding to the init system is provided by a simple init script.

Auxiliary scripts

The auxiliary scripts are stored in /etc/iptables.sh and /etc/ip6tables.sh respectively. They load the basic ruleset that’s common for all servers and after that use run-parts to load the rule fragments:

# INSERT YOUR COMMON IPTABLES RULES HERE 

# This loads all scripts in /etc/iptables.d named *.iptables. No checks are in 
# place, so make sure this directory does not contain anything funky
run-parts --regex '^.*\.iptables' $SCRIPTDIR

Init script

The init script is very simple wrapper that manages iptables.sh and ip6tables. It’s tested on Debian/Ubuntu, but probably works on other LSB-compliant distributions, too.

#! /bin/sh
### BEGIN INIT INFO
# Provides:          iptables
# Required-Start:    $network
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Load iptables ruleset
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin

. /lib/init/vars.sh
. /lib/lsb/init-functions

case "$1" in
    start)
    /etc/iptables.sh
    /etc/ip6tables.sh
        ;;
    restart|reload|force-reload)
    /etc/iptables.sh
    /etc/ip6tables.sh
        ;;
    stop)
    # I've used the -f switch to load a failsafe ruleset. 
    /etc/iptables.sh -f
    /etc/ip6tables.sh -f
        ;;
    *)
        echo "Usage: $0 start|stop" >&2
        exit 3
        ;;
esac

Core iptables module

This is the core iptables puppet module stored in $confdir/modules/iptables/manifests/init.pp.

class iptables::install {
    package { "iptables":
        ensure => present
    }
}

class iptables::config {

    # Defaults for all files defined in this class
    File {
        require => Class["iptables::install"],
        owner => root,
        group => root,
        mode => 755,
    }

    # Make sure the rule directories exist
        file { [ "/etc/iptables.d", "/etc/ip6tables.d" ]:
                ensure => directory,
        }

    # Install an init.d script which takes care of calling the actual 
    # rule-loading scripts at boot time
    file { "iptables":
        ensure => present,
        name  => "/etc/init.d/iptables",
        source => "puppet:///iptables/iptables",
    }

    # Use a preconfigured firewall core on each client. Puppet modules may 
    # add new firewall rules by placing them to /etc/iptables.d/. The 
    # iptables.sh script served by this module then loads them 
    # automatically.
    file { "iptables.sh":
        ensure => present,
        name  => "/etc/iptables.sh",
        source => "puppet:///iptables/iptables.sh",
    }

    # Same for IPv6
    file { "ip6tables.sh":
        ensure => present,
        name  => "/etc/ip6tables.sh",
        source => "puppet:///iptables/ip6tables.sh",
    }

    # This could probably replaced with
    # 
    #   Notify => Service["iptables"]
    #
    # if the init script was little more complex

    exec { "iptables-core-refresh":
        command => "/etc/init.d/iptables restart",
                cwd => "/etc/init.d",
                path => "/sbin",
        refreshonly => true,
        subscribe => [ File["iptables.sh"], File["ip6tables.sh"] ],
    }

}

class iptables::service {

    service { "iptables":
        enable => true,
        require => Class["iptables::config"],
    }
}

class iptables {
    include iptables::install, iptables::config, iptables::service
}