Creating Virtual Networks with KVM on Gentoo

Sometimes you need several virtual machines that are able to communicate with each other and the internet for testing purposes. Today I want to show how this can be achieved on a gentoo linux box using KVM.

Kernel-based Virtual Machine (KVM) is a virtualization solution for the Linux Kernel. It provides the /dev/kvm interface which is used by qemu to emulate guests. KVM can only be used if your CPU supports the Vt-x (Intel) or AMD-V (AMD) extensions. These Extensions enable emulation at almost native speed. Also it is possible to use the vhost-net kernel module to let the network traffic run in its own kernel threads.

Prerequisites

Before we start, I assume that you have a gentoo box up and running. If you do not know how to set up Gentoo Linux you should start with the great documentation. You can check wheather your CPU supports KVM with the following command. If you see the highlighted extension you are all good to start.

                
                      egrep --color -E "vmx|svm" /proc/cpuinfo

Host setup

Kernel setup

As KVM works in kernel space you need to compile the corresponding modules. After invoking menuconfig you need to activate the “Virtualization” category

[*] Virtualization

Also you need to activate support for your cpu and for vhost-net. The following example would fit for an Intel CPU.

[*] Virtualization
--->
    <*>   Kernel-based Virtual Machine (KVM) support
    <*>     KVM for Intel processors support
    < >     KVM for AMD processors support
    <M>   Host kernel accelerator for virtio net

Finally there are some Kernel modules needed by libvirt

General Setup
--->
    [*] Control Group support
    --->
        [*]   Freezer cgroup subsystem
        [*]   Device controller for cgroups
        [*]   Group CPU scheduler
        [*]   Enable perf_event per-cpu per-container group (cgroup) monitoring
        [*]   Block IO controller
        [*]   Resource counters
        [*]     Memory Resource Controller for Control Groups (NEW)
        [*]       Memory Resource Controller Swap Extension (NEW)
        [*]       Memory Resource Controller Kernel Memory accounting (EXPERIMENTAL) (NEW)
-*- Networking support
--->
    Networking Options
    --->
        <M> 802.1d Ethernet Bridging
        <*> Network priority cgroup
        [*] QoS and/or fair queueing
        --->
            <M>   Control Group Classifier
        [*] Network packet filtering framework (Netfilter)
        --->
            <M>   Ethernet Bridge tables (ebtables) support
            --->
                <M>   ebt: mark target support (NEW)
Device Drivers
--->
    Character devices
    --->
        -*- Unix98 PTY support
        [*]   Support multiple instances of devpts
    [*] Network device support
    --->
        -*-   Network core driver support
        <M>     Universal TUN/TAP device driver support
Kernel Hacking
--->
    [*] Debug Filesystem

After compiling your brand new kernel your KVM host is ready for the next steps.

QEMU / libvirt setup

The Quick Emulator (QEMU) can work with many virtualization drivers (such as KVM or XEN) or with its own built-in user-space driver. Using QEMU without an accelerator is slow as hell. Luckily we are using KVM, so we can reach almost native speeds. Although QEMU has lots of USE flags, all recommended flags are set by default.

Now we are going to configure libvirt, which is a management tool for various virtualization solutions. It offers us virsh which is a powerful management user interface to control QEMU and our virtual network.

As we want to use the virtual network capabilities and the QEMU support of libvirt, we need to enable the corresponding USE-flag

echo ">=app-emulation/libvirt-1.0.3-r2 virt-network qemu" >> /etc/portage/package.use

We are now ready to install QEMU and libvirt. emerge should output the following

emerge -pv libvirt

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-misc/scrub-2.5.2  158 kB
[ebuild  N     ] net-firewall/ebtables-2.0.10.4  USE="-static" 102 kB
[ebuild  N     ] sys-apps/dmidecode-2.11  USE="(-selinux)" 54 kB
[ebuild  N     ] dev-libs/libdaemon-0.14-r1  USE="-doc -examples -static-libs" 333 kB
[ebuild  N     ] sys-power/pm-quirks-20100619  10 kB
[ebuild  N     ] sys-libs/libseccomp-1.0.1  USE="-static-libs -tools" 127 kB
[ebuild  N     ] sys-firmware/sgabios-0.1_pre8  30 kB
[ebuild  N     ] dev-libs/libaio-0.3.109-r4  USE="(-multilib) -static-libs {-test}" 43 kB
[ebuild  N     ] sys-devel/bin86-0.16.19  148 kB
[ebuild  N     ] dev-util/gperf-3.0.4  961 kB
[ebuild  N     ] net-libs/libssh2-1.4.2  USE="zlib -gcrypt -static-libs {-test}" 665 kB
[ebuild  N     ] app-text/xhtml1-20020801-r4  227 kB
[ebuild  N     ] dev-libs/libnl-3.2.22:3  USE="-doc -static-libs -utils" 714 kB
[ebuild  N     ] dev-libs/nettle-2.6:0/4  USE="gmp -doc -static-libs {-test}" 0 kB
[ebuild  N     ] dev-lang/swig-2.0.9  USE="pcre -ccache -doc" 5,183 kB
[ebuild  N     ] sys-firmware/ipxe-1.0.0_p20130225  USE="qemu -iso -undi -usb -vmware" 2,157 kB
[ebuild  N     ] sys-firmware/seabios-1.7.2.1  USE="binary" 518 kB
[ebuild  N     ] dev-perl/Text-Unidecode-0.40.0  101 kB
[ebuild  N     ] dev-perl/libintl-perl-1.200.0  489 kB
[ebuild  N     ] sys-devel/dev86-0.16.19  697 kB
[ebuild  N     ] net-dns/libidn-1.26  USE="nls -doc -emacs -java -mono -static-libs" 3,322 kB
[ebuild  N     ] x11-libs/libpciaccess-0.13.1  USE="zlib -minimal -static-libs" 345 kB
[ebuild  N     ] sys-libs/libcap-ng-0.6.6  USE="python -static-libs" 359 kB
[ebuild  N     ] net-dns/dnsmasq-2.66  USE="dhcp ipv6 nls -auth-dns -conntrack -dbus -dhcp-tools -idn -lua -script -tftp" LINGUAS="de -es -fi -fr -id -it -no -pl -pt_BR -ro" 392 kB
[ebuild  N     ] net-misc/radvd-1.9.2-r1  USE="(-selinux)" 177 kB
[ebuild  N     ] dev-libs/libtasn1-2.14  USE="-doc -static-libs" 0 kB
[ebuild  N     ] sys-apps/dbus-1.6.12  USE="-X -debug -doc (-selinux) -static-libs -systemd {-test}" 1,889 kB
[ebuild  N     ] sys-libs/libcap-2.22  USE="pam" 59 kB
[ebuild  N     ] sys-firmware/vgabios-0.7a  USE="-debug" 1,474 kB
[ebuild  N     ] dev-perl/Unicode-EastAsianWidth-1.320.0  29 kB
[ebuild  N     ] net-libs/libtirpc-0.2.2-r1  USE="-kerberos -static-libs" 462 kB
[ebuild  N     ] net-libs/gnutls-2.12.23-r1  USE="cxx nettle nls zlib -bindist -doc -examples -guile -lzo -pkcs11 -static-libs {-test}" 0 kB
[ebuild  N     ] app-laptop/radeontool-1.6.3  359 kB
[ebuild  N     ] app-text/texi2html-5.0-r1  USE="unicode" 15,037 kB
[ebuild  N     ] app-emulation/qemu-1.4.2  USE="aio caps curl filecaps jpeg ncurses png python seccomp threads uuid vhost-net vnc -accessibility -alsa -bluetooth -debug -fdt -iscsi -mixemu -opengl -pulseaudio -rbd -sasl -sdl (-selinux) -smartcard -spice -static -static-softmmu -static-user -systemtap -tci {-test} -tls -usbredir -vde -virtfs -xattr -xen -xfs" PYTHON_TARGETS="python2_7 -python2_5 -python2_6" QEMU_SOFTMMU_TARGETS="x86_64 -alpha -arm -cris -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -or32 -ppc -ppc64 -ppcemb -s390x -sh4 -sh4eb -sparc -sparc64 -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="-alpha -arm -armeb -cris -i386 -m68k -microblaze -microblazeel -mips -mipsel -or32 -ppc -ppc64 -ppc64abi32 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -unicore32 -x86_64" 10,176 kB
[ebuild  N     ] sys-power/pm-utils-1.4.1-r2  USE="-alsa -debug -ntp" VIDEO_CARDS="intel radeon" 204 kB
[ebuild  N     ] app-emulation/libvirt-1.0.5.2  USE="caps libvirtd macvtap nls python qemu udev vepa virt-network -audit -avahi -firewalld -fuse -iscsi -lvm -lxc -nfs -numa -openvz -parted -pcap -phyp -policykit -rbd -sasl (-selinux) -systemd -uml -virtualbox -xen" 23,393 kB

If portage complains about netcat blocking netcat6

* Error: The above package list contains packages which cannot be
* installed at the same time on the same system.

(net-analyzer/netcat-110-r9::gentoo, installed) pulled in by
    net-analyzer/netcat required by @selected

(net-analyzer/netcat6-1.0-r2::gentoo, ebuild scheduled for merge) pulled in by
    >=net-analyzer/netcat6-1.0-r2 required by (app-emulation/libvirt-1.0.5.2::gentoo, ebuild scheduled for merge)

you can solve it this way

emerge -qC netcat && emerge netcat6

This uninstalls netcat and installs netcat6 as a drop in replacement.

After you successfully installed libvirt, you can start it with

/etc/init.d/libvirtd start

Setting up a template VM

As a Gentoo installation can take some time, I prefer to create a template and clone it when I need a new machine. First you need to create a volume for your VM. We use virsh to create it

virsh vol-create-as default Vanilla.img 8192M

This example creates a virtual disk with a size of 8GB, which is sufficient for most installs. If you decide to create smaller disks, you need to take care that you create enough inodes while you create the filesystem. This is important as for example a 3GB disk formatted with the standard ext4 options has not enough inodes for a Gentoo installation.

The next steps depend on wheather you want to use a graphical interface or cli interface.

Using cli

You can start an installation with the following command

virt-install \
    --cdrom /var/lib/libvirt/images/install-amd64-minimal-20130725.iso \
    -r 1024 \
    --accelerate \
    -n Vanilla \
    --disk /var/lib/libvirt/images/Vanilla.img \
    --noautoconsole \
    --graphics vnc,listen=0.0.0.0,port=5900 \
    -m 00:00:00:00:00:01 \
    --os-variant=virtio26

This starts the live image located at /var/lib/libvirt/images/install-amd64-minimal-20130725.iso with 1024MB ram. The machine is accelerated by KVM. The name of the VM is Vanilla and it will use the fresh created image /var/lib/libvirt/images/Vanilla.img which has a size of 8GB. The mac is hardcoded to 00:00:00:00:00:01, so we can use it later for the network configuration and the os type is set to newer linux with virtio enabled. Also a vnc server is launched at port 5900. You can connect to this vnc server with tools like vinage or vnc-viewer.

Using virt-manager

If you are on a desktop machine, you can use virt-manager to setup the VM. The process is straight forward. Just start with a click on new and follow the instructions. You can even connect to a remote libvirt daemon and install your VMs using a graphical interface. To install virt-manager just execute

emerge -av virt-manager

If you have trouble starting the virt-manager you might need to switch your python profile.

host ~ # eselect python list
    Available Python interpreters:
      [1]   python2.7
      [2]   python3.2 *

host ~ # eselect python set 1

host ~ # eselect python list
    Available Python interpreters:
      [1]   python2.7 *
      [2]   python3.2

Installation

The Installation process is similar to the one described in the Gentoo handbook with a few differences

Kernel Configuation

  Processor type and features
  --->
      [*] Paravirtualized guest support
      --->
          [*]   KVM paravirtualized clock
          [*]   KVM Guest support

  Device Drivers
  --->
      Virtio drivers
      --->
          <*>   PCI driver for virtio devices
          [*] Block devices
          --->
              <*>   Virtio block driver
      [*] Network device support
      --->
          <*>   Virtio network driver

the hdd is mapped to /dev/vda instead of /dev/sda
the network configuration should be set to dhcp (leave /etc/conf.d/net blank)

Now that we have a working template, we can create a script to automate VM generation.

VM Automation

As I feel most comfortable using perl scripts, the following example is no exception. As I am not using any perl specific libraries this example can be easily adapted to other languages. Note that this script is Gentoo specific (paths etc.) and might not necessarily work on other operating systems.

Note: You need virt-manager to be installed, as virt-clone is part of the package.

clone.pl

                
                  
                    
                    
                      #!/usr/bin/perl

use strict;
use warnings;

use Getopt::Long;
use File::Basename;
use Template;

our $PROGNAME = basename($0);

my $nodeCount;
my $qemuDir = '/etc/libvirt/qemu';

my $networkConf   = 'custom.xml';
my $existingNodes = [];

GetOptions( "n=i" => \$nodeCount, );

unless ( defined($nodeCount) ) {
    print "usage: $PROGNAME --n <number of machines to clone>\n";
    exit(0);
}

# get the highest VM number
sub getCurrentMax {
    my $maxNodeNo = 0;
    opendir( my $dh, $qemuDir ) or die $!;

    while ( my $node = readdir($dh) ) {
        next if ( $node !~ m/VM\d+\.xml$/ );
        if ( $node =~ /\d+/ ) { $maxNodeNo = $& }
    }

    close($dh);
    return $maxNodeNo;
}

# the mac address for the first machine created
sub getInitialMac {
    my $macStr = '00:00:00:00:00:02';
    for ( my $i = 0; $i < getCurrentMax(); $i++ ) {
        $macStr = incrMac($macStr);
    }
    return $macStr;
}

# increment the mac address by 1
sub incrMac {
    my $macStr = shift;
    ( my $macHex = $macStr ) =~ s/://g;
    my ( $macHi, $macLo ) = unpack( " nN ", pack( 'H*', $macHex ) );

    if ( $macLo == 0xFFFFFFFF ) {
        $macHi = ( $macHi + 1 ) & 0xFFFF;
        $macLo = 0;
    }
    else {
        ++$macLo;
    }

    $macHex = sprintf( "%04X%08X ", $macHi, $macLo );
    return join( ':', $macHex =~ /../sg );
}

# write the network.xml file
sub writeVirtNet {
    my $nodeCount = shift;
    my $macStr    = '00:00:00:00:00:02';
    my $tt        = Template->new();
    my $futCount  = $nodeCount + getCurrentMax();
    my $nodes     = [];
    for ( my $i = 0; $i < $futCount; $i++ ) {
        my $node = {
            'ip'   => '192.168.122.' . ( $i + 2 ),
            'name' => 'proxy' .        ( $i + 1 ),
            'mac'  => $macStr,
        };
        $macStr = incrMac($macStr);
        push( @$nodes, $node );
    }

    $tt->process( 'network.xml.tt', { 'hostList' => $nodes },
        $networkConf, );
}

# delete the old default network and load the new one
sub updateNetwork {
    writeVirtNet(shift);
    system("virsh net-destroy default");
    system("virsh net-undefine default");
    system("virsh net-define $networkConf");
    system("virsh net-autostart default");
    system('/etc/init.d/libvirtd restart');
}

updateNetwork($nodeCount);

my $macStr = getInitialMac();

for ( my $i = 0; $i < $nodeCount; $i++ ) {
    my $no = getCurrentMax() + 1;
    # clone the VM
    system(
        "virt-clone --connect=qemu:///system -o Vanilla -n VM$no -f /var/lib/libvirt/images/VM$no.img -m \'$macStr\' "
    );

    # start the VM
    system("virsh --connect=qemu:///system start VM$no");
    $macStr = incrMac($macStr);
}

                    
                  

              

network.xml.tt

                
                  
                    
                    
                      <network>
  <name>default</name>
  <uuid>ff5e37ba-9646-4f4a-8ec9-e393d0d91d66</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:67:08:55'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
[% FOREACH entry = hostList -%]
        <host mac='[% entry.mac %]' name='[% entry.name %]' ip='[% entry.ip %]' />
[% END -%]
    </dhcp>
  </ip>
</network>

                    
                  

              

To create a new VM you just need to call

/usr/bin/perl clone.pl -n 1

This creates a new VM called VM1 with the mac address 00:00:00:00:00:02. Also it updates the default libvirt network to assign the ip address 192.168.122.2 to this VM. To do so it loads the network.xml.tt template and processes it. Another call to this script creates VM2 with the ip address 192.168.122.3 and so on. The resulting machines can communicate with each other, with the host and with the internet.

Conclusion

Setting up a virtual network with KVM on Gentoo is no rocket science. You just have to enable the necessary kernel modules and install libvirt with the virt-network USE-flag. Most of the other stuff works out of the box and can be easily customized to your special needs and automated.

Creating Virtual Networks With KVM on Gentoo