WWWTech

Open Source Software and publications by Christian Kruse

Save Ukraine

Automatic configuration deployment with cdist

Christian Kruse,

In my job as „the IT guy“ at a company in Ettlingen I am also responsible for the company network and the configuration and maintenance of the server infrastructure. In the past I did all this by hand: I configured every service directly on the server. Every configuration change I did with a text editor directly on the server.

While with this approach one is able to react very fast on problems, on the other hand it is error-prone. You can't test changes, you don't have a documentation. And if you have a documentation, it regularly lacks of the latest changes.

If you ever have to migrate to another hardware, you have to copy the right configuration files to the right places and change the right things to adopt it to the new environment. You also have to install the right software dependencies of the software running on the server. In an emergency case a really bad starting situation.

When I recently replaced some old boxes by newer ones to avoid hardware failure, I took the chance and decided to invest some time to automate configuration of the new boxes. The world of automated configuration deployment is rather small. There are just a few solutions, the most often mentioned are Puppet, Chef as well as CFEngine. For my purposes (a rather small company network) the three are too big, too complex. I was looking for something less „magical,“ something following the KISS principle. After a lot of research some friends of mine suggested cdist. cdist has been written by Nico Schottelius, a FOSS hacker and system administrator living in Zurich. It is free software (licensed as GLPv3). cdist configuration deployment consists basically of executing shell scripts which copy or modify files and install packages. Pretty easy, for example this file adds the PostgreSQL repository for Debian to the sources.list:

__block sources-list \
        --state present \
        --file /etc/apt/sources.list \
        --text - <<EOF
deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main
EOF

The __block call is a function defined by cdist, it calls this kind of construct types. Everything you do is calling these types to manipulate, copy, link and move files.

A cdist configurations entry point is the manifest/init file. In this entry point file you specify what to do, an easy example:

__package apache2 --state present

This would install the apache2 webserver packet on the target host.

Configuration deployment by service types

One of the requirements I had was the ability to quickly move a service to another machine, for example because of a hardware failure. So my first approach was to define service functions and define via environment variable the type of service I'd like to deploy. This way I could for example call

SERVICES="web mail" cdist config -c ./conf/ 10.0.0.2

and deploy the webserver configuration and its dependencies as well as the mailserver configuration and its dependencies to the node 10.0.0.2. But this way I've got the problem again that it is not documented when a service moves to another host, and it is error-prone as well: what if I type a service name wrong or add a service name which doesn't belong to this host?

Configuration deployment by host definitions

So I went to a different type of configuration deployment: host definitions. I define in an external file hosts and specify (among some other things) the services to deploy:

SERVICES_10002="web mail"

When I move a service to another host I change this file and make a new git commit; this way every change is documented via git history. It also is less error-prone, since I don't have to specify the services again and again on the command line - I'm sure one day I would forget a service or add a service to the wrong box. And due to this structure my manifest/init file is pretty easy:

for file in conf/manifest/*.sh; do
    . $file
done

This part reads every .sh file containing the modules for the services.

__file /etc/cdist-configured
__cdistmarker

HOSTCONFIG="$(getmap "SERVICES_" "$__target_host")"
HOSTNAME="$(getmap "HOSTNAME_" "$__target_host")"

Since Bash below v4 doesn't support dictionaries, we have to do a little trick to get the configuration for the host. We call a self-defined function getmap, this function basically we constructs a string of the form <first arg><second arg>, gets the value of the variable with that name and returns it to the caller. Thus we read the modules to activate and the hostname we should set.

MY_OS="$(cat "$__global/explorer/os")"
if [ "$MY_OS" = "freebsd" ]; then
    base_freebsd
else
    base_linux
fi

This snippet calls a function which does the basic configuration of the system (hostname, time zone, etc, pp) depending on the target OS.

fun="base_$HOSTNAME"
fn_exists "$fun" && eval "$fun"

This snippet checks if a host-specific configuration function exists, for example to install the RAID monitoring utilities on the right host. If it exists, the function gets called.

for feature in $HOSTCONFIG; do
    if [ fn_exists "$feature" ];
        eval "$feature"
    else
        echo "$feature does not exist!"
        exit 1
    fi
done

This snippet loops to each service defined by the host configuration and calls the function configuring the service. That's it, now we have a host configuration based configuration deployment. Of course, the functions which deploy the actual services still are missing ;-)

Deploying a service configuration

I moved the configuration of a specific service to a function. This way I get a basic modularization; for example the function webserver installs all software requirements and the configuration for the software on the target host. I will try to explain some things with the configuration function for the web server:

webserver() {
    __user ssh_data_in \
           --state present \
           --create-home
    __user git \
           --state present \
           --create-home

This snippet ensures that the users ssh_data_in and git exist, with created home directories.

    require="__user/ssh_data_in" __directory /home/ssh_data_in/.ssh/ \
                --owner ssh_data_in \
                --group ssh_data_in \
                --mode 0700
    require="__directory/home/ssh_data_in/.ssh/" __file /home/ssh_data_in/.ssh/authorized_keys \
           --state present \
           --owner ssh_data_in \
           --group ssh_data_in \
           --source - <<EOF
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIB+bxdCSP8zo04Xmf+ItBlJtgXWhbBP69HetzSj/IH18ugnKrflylDL0SaCAizjKJKZnjnkW0RNqKEHQzMsMSOs8bSfu9L2W5Qg3peJ4T4BZy3Q7cfYNOYkMM6vYl3pv2coJvDlnUc2/BY95NlVHYhW4YtHYPjBTwB3a4eaotl4RQ==
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDosf/bRU/avCgY47OgIY3thRM5cLejAicnghRrjeqzfCR8WGvkaJeKxGc9B+4O4DX3/kp3USOZNzD45Uk6s7LYPRTMWoY+CPgiyuC2fNYmON+lpsnl3ebmY0wyigOIH6HiKf3GZlsnp6E1efxbLlgNBIbyhp3MSs6LeNok+/3090LNdEOJvwOpEHYveJ/qCeHT2ipbzToZS5LGMJ192xqYik/4Ov8drg3WHhusYw1HJl/CM/jE3XPZScCaCGmvGvetsa4nJiy2PBn7Qnp3AYckHw3RyyGShmJb/IfeXI8ypKTqtzmVOZ4RnZk+OS0q/5/o/9pJBg6TB9OuJgBhah+V ckruse@vali.local
EOF

The feature I use in this block is a dependency. cdist does not execute the types/functions in the order I call them, but it may generate some remote code and executes it later. With the special viarable require you can specify dependencies; in this case I want to ensure that the directory /home/ssh_data_in/.ssh/ exists and belongs to the user and group ssh_data_in - but after the user has been created. When the directory has been created we create a file authorized_keys containing two public keys.

    __package libapache2-mod-xsendfile \
              --state present
    __package rsync \
              --state present
    __package syslog-ng \
              --state present
    __package build-essential \
              --state present
    __package ruby1.9.3 \
              --state present
    __package ruby-dev \
              --state present
    __package curl \
              --state present
    __package libcurl4-gnutls-dev \
              --state present
    __package libxml2-dev \
              --state present
    require="__package_update_index" __package libpq-dev \
              --state present
    require="__package_update_index" __package nodejs \
              --state present
    __package php-mail \
              --state present
    __package php-mail-mime \
              --state present
    require="__package/nodejs __package/curl" __install_npm
    __package apache2 \
              --state present
    require="__package/apache2 __package/ruby1.9.3" __package libapache2-mod-passenger \
           --state present

    __package php5 \
              --state present
    require="__package/apache2 __package/php5" __package libapache2-mod-php5 \
           --state present

    require="__package/php5" __package phpldapadmin \
           --state present
    require="__package/php5" __package phppgadmin \
           --state present
    require="__package/php5" __package phpmyadmin \
           --state present

    __package pdftk --state present
    __package git --state present
    require="__package/ruby1.9.3" __package_rubygem bundler --state present

This snippet installs a lot of software packets we need for the software running on the web server. __install_npm is a self-written function installing NPM via node.js.

    require_for_restart="__package/apache2 __package/libapache2-mod-passenger __package/libapache2-mod-php5 __package/libapache2-mod-xsendfile"

    for site in ./files/webserver/apache/sites/*; do
        site_name="`basename $site`"
        doc_root="`grep DocumentRoot $site | sed 's/DocumentRoot//' | tr -d ' \t'`"
        require_for_restart="$require_for_restart __file/etc/apache2/sites-available/$site_name __directory/$doc_root"

        require="__package/apache2" __file /etc/apache2/sites-available/$site_name \
               --source $site \
               --state present

        # if it is a ruby thing we use my user as owner
        if [[ $doc_root == *public* ]]; then
            __directory $doc_root \
                        --state present \
                        --owner ckruse \
                        --group ckruse \
                        --parents
            __directory ${doc_root%/}/.. \
                        --state present \
                        --owner ckruse \
                        --group ckruse \
                        --parents
        else
            __directory $doc_root \
                        --state present \
                        --owner www-data \
                        --group www-data \
                        --parents
            __directory ${doc_root%/}/.. \
                        --state present \
                        --owner www-data \
                        --group www-data \
                        --parents
        fi

        if [ "$site_name" == "default" ]; then
            require_for_restart="$require_for_restart __link/etc/apache2/sites-enabled/000-$site_name"
            require="__file/etc/apache2/sites-available/$site_name" __link /etc/apache2/sites-enabled/000-$site_name \
                   --source ../sites-available/$site_name \
                   --type symbolic
        else
            require_for_restart="$require_for_restart __link/etc/apache2/sites-enabled/$site_name"
            require="__file/etc/apache2/sites-available/$site_name" __link /etc/apache2/sites-enabled/$site_name \
                   --source ../sites-available/$site_name \
                   --type symbolic \
                   --state present
        fi
    done

    require="__package/apache2" __file /etc/apache2/sites-available/default-ssl \
           --state absent
    require="__package/apache2" __link /etc/apache2/sites-enabled/default-ssl \
           --source ../sites-available/default-ssl \
           --type symbolic \
           --state absent
    require="__package/apache2" __link /etc/apache2/mods-enabled/ssl \
           --source ../mods-available/ssl \
           --type symbolic \
           --state absent
    require="__package/apache2" __link /etc/apache2/mods-enabled/expires.load \
           --source ../mods-available/expires.load \
           --type symbolic \
           --state present
    require="__package/apache2" __link /etc/apache2/mods-enabled/headers.load \
           --source ../mods-available/headers.load \
           --type symbolic \
           --state present
    require="__package/apache2" __link /etc/apache2/mods-enabled/xsendfile.load \
           --source ../mods-available/xsendfile.load \
           --type symbolic \
           --state present
    require="__package/apache2" __file /etc/apache2/mods-available/xsendfile.conf \
           --state present \
           --source <<EOF
XSendFile On
XSendFilePath /mnt/storage/noodles/
EOF
    require="__file/etc/apache2/mods-available/xsendfile.conf" __link /etc/apache2/mods-enabled/xsendfile.conf \
           --source ../mods-available/xsendfile.conf \
           --type symbolic \
           --state present

    require_for_restart="$require_for_restart __file/etc/apache2/sites-available/default-ssl __link/etc/apache2/sites-enabled/default-ssl __link/etc/apache2/mods-enabled/ssl __link/etc/apache2/mods-enabled/xsendfile.conf"

    for to_enable in passenger.load passenger.conf rewrite.load; do
        require_for_restart="$require_for_restart __link/etc/apache2/mods-enabled/$to_enable"
        require="__package/apache2" __link /etc/apache2/mods-enabled/$to_enable \
               --source ../mods-available/$to_enable \
               --type symbolic \
               --state present
    done

    require_for_restart="$require_for_restart __block/apache2-charset"
    require="__package/apache2" __block apache2-charset \
            --state present \
            --file /etc/apache2/conf.d/charset \
            --text - <<EOF
AddDefaultCharset UTF-8
EOF

    require_for_restart="$require_for_restart __remove/etc/apache2/conf.d/phpldapadmin"
    require="__package/phpldapadmin" __remove /etc/apache2/conf.d/phpldapadmin \
           --pattern 'Alias /phpldapadmin'

    require_for_restart="$require_for_restart __remove/etc/apache2/conf.d/phppgadmin"
    require="__package/phppgadmin" __remove  /etc/apache2/conf.d/phppgadmin \
           --pattern "Alias /phppgadmin"

    require="$require_for_restart" __service apache2 --action restart
}

This block configures the Apache web server, enables some modules and installs some virtual hosts. It also ensures that some files don't exist. For example we use a NGINX for SSL termination so we don't want the SSL module of Apache to be enabled. When all files and packages have been installed, we restart the service. And bam, we have a working web server setup! Of course there is a little bit more in the real deployment function, I shortened it a bit to the essential things.

Conclusion

cdist is a nice and simple tool. The complete company infrastructure now gets configured using this approach. Yay!