Rex in Practice: Test-driven Infrastructure

I hate writing tests. There’s only one thing I hate more: not having tests. So I like it when writing and running tests for anything is easy.

In part 1 of Rex in practice series, we got started with describing our infrastructure as code. All of those automation bits are kept in git repositories. They are nothing but code after all. Since they are code, we want them covered by tests.

Normally, we would start with writing tests which can check for the expected state of a remote machine, and then we write our code in iterations to pass all the cases.

Rex supports managing virtual machines and containers through different methods, like LibVirt, VirtualBox or Docker (and even some cloud providers). Built on top of this functionality, it also has a Vagrant-like feature called Rex::Box.

Rex::Test in turn, makes use of Rex::Box to quickly create a VM, provision it by running one or more tasks and then run a series of tests checking the state of the machine.

Following up on the example in the previous part of the series, our NTP tests would probably be similar to this:

t/ntp.t
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
use Rex::Test::Base;
use Rex::Commands::MD5;

set box => 'KVM';

my $test_vm = Rex::Test::Base->new;

$test_vm->name('ntp_test');
$test_vm->base_vm('https://images.domain.tld/base_image.qcow2');
$test_vm->vm_auth( user => 'root' );

$test_vm->run_task('ntp');

$test_vm->has_package('ntp');

$test_vm->has_file('/etc/ntp.conf');
$test_vm->has_stat('/etc/ntp.conf', { owner => 'root', group => 'root' });
$test_vm->has_content('/etc/ntp.conf', qr{server /d.gentoo.pool.ntp.org} );

my $checksum = md5 '/etc/ntp.conf';
$test_vm->ok( $checksum eq '9c35e8b49506857b244872291cc9f12a', '/etc/ntp.conf MD5 checksum is correct' );

$test_vm->has_service_running('ntpd');

$test_vm->finish;

1;

Let’s see the elementary steps of this example:

1
2
use Rex::Test::Base;
use Rex::Commands::MD5;

It starts with importing some modules we would like to use: Rex::Test::Base for the tests themselves and Rex::Commands::MD5 for the md5 command used later in one of the examples.

1
set box => 'KVM';

The default virtualization method is VirtualBox, but in this example we would like to use a KVM box, so we need to specify that explicitly.

1
my $test_vm = Rex::Test::Base->new;

Next we instantiate a Rex::Test::Base object, called test_vm. Its methods will enable us to tell Rex what we would like to test and how.

1
2
3
$test_vm->name('ntp_test');
$test_vm->base_vm('https://images.domain.tld/base_image.qcow2');
$test_vm->vm_auth( user => 'root' );

First we give a name to the VM which will run the tests (ntp_test in this case), then point Rex to the base image to use when creating this new VM, and finally specify the authentication credentials for the VM.

Rex downloads the specified image into ./tmp and then tries to import it as a new VM, cloning the base image into ./storage (so the original file is left untouched and can be reused multiple times). Depending on the virtualization method requested and the type of the image, Rex also tries to extract and/or convert it before using it. For example if the specified base image is a .gz file or if a file in OVA format should be used with KVM.

1
$test_vm->run_task('ntp');

As the last step of initialization, this line will provision the VM by running a Rex task called ntp on it. That’s the task we specified in the previous post, but normally would define it when this step fails. It is possible to run multiple tasks, by passing them as an array reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$test_vm->has_package('ntp');

$test_vm->has_file('/etc/ntp.conf');
$test_vm->has_stat('/etc/ntp.conf', { owner => 'root', group => 'root' });
$test_vm->has_content('/etc/ntp.conf', qr{server /d.gentoo.pool.ntp.org} );

my $checksum = md5 '/etc/ntp.conf';
$test_vm->ok( $checksum eq '9c35e8b49506857b244872291cc9f12a', '/etc/ntp.conf MD5 checksum is correct' );

$test_vm->has_service_running('ntpd');

$test_vm->finish;

1;

After the boilerplate and test initialization, let’s see the tests themselves. The above code snippet executes the following tests inside the VM in order:

  • check if it has a specific package installed, called ntp
  • check if it has a specific file present, called /etc/ntp.conf
  • check if that file has specific properties, like owner and group
  • check if that file has a specific content, matching the regular expression server /d.gentoo.pool.ntp.org
  • run the md5 Rex command inside the VM to calculate the MD5 checksum for the same configuration file, and then check if it matches a specific value
  • check if the VM has a specific service (ntpd) in a running state
  • finish the test suite

As you can see, we’re not limited to the built-in tests, but we can run arbitrary commands inside the VM, record their output or return code, and then check them against their expected values with the ok() method.

Before we can actually run the test via Rex, we need to add one more line to our Rexfile we showed in the previous post:

Rexfile
1
2
3
4
use Rex -feature => [ '1.2' ];
+use Rex::Test;

user 'root';

This enables an internal Rex task, called Test:run, which by default runs all test cases under the ./t directory:

1
$ rex Test:run

If we had more tests there, we could pick only one or few of them to be run:

1
2
$ rex Test:run --test=t/ntp.t
$ rex Test:run --test=t/ntp*

When running a test, Rex outputs its current progress and of course each of the test results, plus an overall result like how many tests were run, and whether the test suite failed or passed. For example something like this (note the -q command line option to make Rex output quiet):

1
2
3
4
5
6
7
8
9
10
11
$ rex -q Test:run
Waiting for SSH to come up on 192.168.122.206:22.
ok 1 - Found package ntp
ok 2 - Found /etc/ntp.conf file.
ok 3 - Owner of /etc/ntp.conf is root
ok 4 - Group of /etc/ntp.conf is root
ok 5 - Content of /etc/ntp.conf contain (?^:server \d.gentoo.pool.ntp.org).
ok 6 - /etc/ntp.conf MD5 checksum is correct
ok 7 - Service ntpd running.
1..7
PASS

About the author

Ferenc Erki Ferenc Erki is a core developer of Rex and a system administrator at adjust, where he is known as tamer of the ELK beast.

Comments