At adjust we use (R)?ex extensively to automate tasks related to our infrastructure, and we also started to use it for application deployment.

We would like to share our use case with this tool, highlighting some of its features through a series of introductory posts and examples.

Introduction to Rex

Rex is a deployment and configuration management framework written in Perl, which uses SSH to manage remote hosts.

Since nothing else is needed for the core functionality, chances are high that you can just start using it right away, regardless of whether you would like to do the management from a machine running Linux, Mac OS X, Windows or practically anything that can run Perl code.

Using SSH as a transport layer means that solutions to problems like authentication and encryption are simply reused, allowing Rex to focus on automation. It also enables Rex to manage a bit more exotic remote machines such as OpenWRT boxes or even iDRAC interfaces (with Windows management support on the roadmap).

Rex provides a simple DSL to easily describe the steps you would like to automate, but in the end everything is just plain Perl, so you are free to harness its full power if needed. If you are not familiar with Perl and would like to get a quick introduction on the basics, check out Rex authors’ Just enough Perl for Rex page.

While Rex is primarily used as a push-style configuration management tool, as it is usual with Perl, there is more than one way to do it (TIMTOWTDI). For example:

• you can do pull-style management by periodically downloading task definitions from a git repository or a web server, and then running them locally (by the way, official server and agent are also on the roadmap, if that is your fancy)
• if you need to scale to more than a couple of hundred remote servers, you can distribute job execution e.g. via Gearman (even with optional queueing to have deferred execution in case a remote is down for a while)

As you can see flexibility is a key design concept for Rex and it lets you solve your own problems in your own style without getting too much in your way.

A simple example

Whether you call yourself a software developer or system administrator (which are less and less distinct, by the way), you are most probably providing services to customers. I mean customer as in: any end user of any service is a customer. In general the lifecycle of those services has the following three common tasks:

1. installing
2. configuring
3. running

Of course we can extend it with upgrading, monitoring and uninstalling, but for the sake of simplicity let’s focus on the previous list for now and see an example of how Rex deals with them.

At the core of any automation project based on Rex there is a Rexfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

use Rex -feature => [ '1.2' ];

user 'root';

group servers => 'server-[1..12].domain.tld';

environment demo => sub {
    group servers => 'demo-[1,2].domain.tld';
};

desc 'Setup NTP';
task 'ntp', group => 'servers', sub {
    pkg 'ntp', ensure => 'present';

    file '/etc/ntp.conf',
        ensure    => 'present',
        source    => 'files/etc/ntp.conf',
        owner     => 'root',
        group     => 'root',
        mode      => '0644',
        on_change => sub { service ntpd => 'restart' };

    service ntpd => ensure => 'running';
};

A Rexfile has three main parts: authentication details, configuration options and task definitions. Let’s see the details of this example:

1

use Rex -feature => [ '1.2' ];

First we import Rex and enable the feature flag for version 1.2.0.

1

user 'root';

In order to connect to a remote machine, we need to specify the credentials to be used during authentication. Having only a user specified is the most simple case (while using the default SSH provider on Unix-like systems and an SSH agent). Of course, there are many ways to authenticate for a remote system, but instead of giving a boring list of those options here, we’d like to point the reader to further resources on this topic:

• Authentication chapter of the work-in-progress Rex Book.
• Levels of security using (R)?ex article by Gabor Szabo.
• Rex::Commands API documentation on how to override authentication options only for a given group or task with auth for.

1

group servers => 'server-[1..12].domain.tld';

The next step is to define which servers and server groups we have. Our example will generate a server group called servers with 12 hosts in it: server-1.domain.tld, server-2.domain.tld, ..., server-12.domain.tld.

Optionally, those server group definitions can come from external sources like INI or XML files, SQL queries, Nagios configuration, etc. Or, since it is nothing more than plain Perl, any array can be passed to group to be used as a list of servers.

1
2
3

environment demo => sub {
    group servers => 'demo-[1,2].domain.tld';
};

If you have several environments to manage - like testing, staging, qa, demo, production, and so on - you can easily override group definitions or authentication options for these environments. Or even define tasks that are only available for these specific environments. Later on, you can choose to run a task on only one of the environments with rex -E demo ... on the command line.

1

desc 'Setup NTP';

With desc we give our following task a nice description to be shown, e.g. in the task list printed by rex -T.

1
2
3
4
5
6
7
8
9
10
11
12
13

task 'ntp', group => 'servers', sub {
    pkg 'ntp', ensure => 'present';

    file '/etc/ntp.conf',
        ensure    => 'present',
        source    => 'files/etc/ntp.conf',
        owner     => 'root',
        group     => 'root',
        mode      => '0644',
        on_change => sub { service ntpd => 'restart' };

    service ntpd => ensure => 'running';
};

Let’s go on to the most interesting part: the task definition itself. In our example we define a task called ntp, and associate it with the server group called servers by default.

Within the task itself, we specify the main lifecycle steps of the NTP service:

1. install the package called ntp
2. configure NTP by copying the local files/etc/ntp.conf to /etc/ntp.conf on the remotes and ensuring proper owner/group/mode properties for it, plus restarting the ntpd service on the remotes where the configuration file in question has changed
3. ensure that the ntpd service is running and will be started after reboot

That’s it. Three steps, three commands.

Please also note that this code doesn’t assume any specific OS on the remotes (well, other than the package and service names). It’s the job of Rex to figure that out and use the proper package or service management methods.

Given that Rexfile, it takes only a single command to setup, configure and run NTP on all twelve servers:

1

$ rex ntp

Or the same but using the demo environment:

1

$ rex -E demo ntp

Since the Rexfile is nothing but code, it makes sense to include it in a version control system such as git and share it with your coworkers, so they also can start using and improving it. There you go, from zero to infrastructure as code in a few easy steps.

About the author

Ferenc Erki is a core developer of Rex and a system administrator at adjust, where he is known as tamer of the ELK beast.