Virtual Laboratory with Linux, KVM, etc.

(April 2018)

We want to create an isolated virtual test environment, where it’s easy to spin-up virtual machines.

Install any necessary packages for basic KVM virtualization, including kvm, qemu-kvm, and libvirt-clients on Debian, or libvirt-daemon-kvm, qemu-kvm, and libvirt-client on RHEL.

To run the VM as a non-root user, this may be necessary:

# usermod -a -G kvm paulgorman

Create a Network Bridge

A Linux bridge device acts like a virtual network switch. Even if we already have a bridge for routine use (i.e., for use with KVM or Docker), create a new bridge for the isolated test lab. It may be useful to install the bridge-utils package.

# ip link add lab-br0 type bridge

Possibly needed:

# chmod 4755 /usr/lib/qemu/qemu-bridge-helper

Prepare Cloud-init

In order to conserve disk space, and make instance create fast and easy, we’ll use QEMU’s ability to make new sparse disk images based on existing qcow2 images. Install the qemu-tools package on Debian or qemu-img on RHEL. Download a qcow2 base image to use for instances. It’s important to use a particular version for the base image, since replacing the base image with the “latest” version would corrupt images derived from the previous base.

$ wget https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1802.qcow2.xz
$ tar xf CentOS-7-x86_64-GenericCloud-1802.qcow2.xz
# mv CentOS-7-x86_64-GenericCloud-1802.qcow2 /var/lib/libvirt/images/

Because all our instances start from the same vendor-supplied base image, we use cloud-init to customize them for our needs. Minimally, we install a key so we can SSH into the instances. If we care about setting unique hostnames, we’ll generate a new meta-data file for each instance and recreate the cidata.vfat image; otherwise, reuse the image for every instance.

$ mkdir -p ~/libvirt/cidata

$ cat > ~/libvirt/cidata/meta-data <<EOF
instance-id: iid-012345
local-hostname: myvm-CentOS-7-x86_64
EOF

$ cat > ~/libvirt/cidata/user-data <<EOF
#cloud-config
timezone: America/Detroit
locale: en_US.UTF-8
users:
  - name: paul
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash
    ssh-authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqia4ScZ+v6Xuvv4HrTf8R4wIHAilul7LIlR5LTKzHvaCXrTAVkIqUh24cJMEXCOKyS7kOnAkbNzKj/NPQNfw42JQzmGF3I6VT8HTqi2NfTQ4lqe+vaMiCZU9mW8q33J3SJzLD/HciA2SqqSr3QMI9urDLn3ovpyKqYjxnSo1f9i+LInmXGgFfmuWbGAXpWJynUJxNB0WzZa3NpCmexlXw1f27tR1+fSJUDDP/HfwimmZms7Q8hA03dohI5OJmbI2YAX+OrHfO4Bya1jeQ5vLKoEX7jywLXHhuJORJ+6fUZBGL6sMQGfWKbYThdpU8MDmNSlOdHl9incoS8b9Q8trJ
EOF

$ virt-make-fs --type=vfat --label=cidata ~/libvirt/cidata ~/libvirt/cidata.vfat

We can reuse this cidata.vfat image for ever instance we create (unless we care about setting different hostnames or usernames on initial boot).

I have run into problems getting the Debian OpenStack qcow2 image to do nocloud cloud-init against a vfat image. For debian, I use the cloud-localds utility (from the cloud-image-utils package) instead (which, I think, uses genisoimage):

$ cloud-localds -v ~/libvirt/seed.img ~/libvirt/cidata/user-data.yaml ~/libvirt/cidata/meta-data.yaml

Create a New Instance

#!/bin/sh
set -euf

d=$(date +%Y%m%d%H%M%S)
vm=lab-"$d"-CentOS-7

echo "Set VM name to $vm"

cat > ~/libvirt/cidata/meta-data <<EOF
instance-id: iid-$d
local-hostname: $vm
EOF

virt-make-fs --type=vfat --label=cidata ~/libvirt/cidata ~/libvirt/cidata.vfat

qemu-img create -f qcow2 -b /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud-1802.qcow2 ~/libvirt/"$vm".qcow2 10G

virt-install --import \
        --name="$vm" \
        --ram=1024 \
        --vcpus=1 \
        --network bridge=lab-br0 \
        --os-variant rhel7.4 \
        --graphics none \
        --disk ~/libvirt/"$vm".qcow2 \
        --disk ~/libvirt/cidata.vfat

virsh detach-disk --persistent "$vm" vdb

Press Ctrl-] to disconnect from the instance console.

Delete an instance:

#!/bin/sh
set -euf

virsh destroy $1
virsh undefine $1
rm ~/libvirt/"$1".qcow2

Create a Debian instance:

#!/bin/sh
set -euf

d=$(date +%Y%m%d%H%m%S)
vm=asterisk-"$d"-Debian-9

echo "VM name set to $vm"

cat > ~/libvirt/cidata/meta-data <<EOF
instance-id: iid-$d
local-hostname: $vm
EOF

cat > ~/libvirt/cidata/user-data <<EOF
#cloud-config
users:
  - name: paul
    groups: sudo
    shell: /bin/bash
    sudo: ['all=(all) nopasswd:all']
    ssh-authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuGq7I5AhEs9EcZ0sbHpo0fSYZczjrfZcFBsH0ue/S7wcmvTBNWc2t9vy8SpT+HwRngeLYSgBkWcOxwrWvQlXY9e4E9m2/CAQu57X2gI5NhiiCLHloqQ4c+kJmgr4fHbAPtgqHIpT7Bzd5CmKn0IFyzlDr9VpyDtCNeRk44Ery80yfMkoKVOzlIHKct5TeTfnZPjtgKoLJkgYg6+G9x6QWOmfL1l5Bj7WKZcC57jQsf+gxFnwZrfdSaCCg3h4F+F0KlkYvGjE3c5CJY8je41sMVKkoC9ujfZNdT0n3rNb9KTlwv+nPICFIK8gn3hwjNcKWyKs0qw5CZHDZiZNpXegIWhJdIAE0X1txDtXYJ7yM/sO3U3pa8DR0AoyY/3TgML8Y/fpqk4nWDi/s/7dDRlbNaD6pi5xYe+qeYJUhw77B39jKWHn8o+1IPzuulqNF3A0xC5+lXoTaXWIiNVv+sZB6nZkjrWuW6NcMFJdFQ2G6t4K/+9TcOUygh95/Ekx7iGN9uw1G4Xv5eRHSb7bJbi7Q2z5ZTh0i2XOYqp5aHtcm7Ok9VzzvScr8TRgp4hafh5IWoABWNTedod0a9spaYEZo9lpQk7UekY37tucqj81N9wramCbYecCLjUhMSpr9/SLDgHAzId1XoaZ99PId4g1FLOms7u11U3bTTW4BGbP4tw==
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDWc7JmBmB3hZuyU48PXsJhWR4BKM0ILyj4esC47r8xKBzd2ZfFQtKaM0ZLPbEcYEYcpWHB7BlIohRhtAZl7gblyGxd9M3e4tfHXea8j1b0OsdImNw5IlUEET3MsdlcUY8OmFud6y8bMoAfdep4mQf5CcEcod5uJyoNOCO1LHR4coAH+KEWhV2pzQHm+AI6Kwcmj4BflAHg4b09sDzPlrLgT//nKtqkOtJW7T5FqzbrmCksEF6hW/wq9Ov4Mmc0UIclsAHXIrL4UW9o8JEwPmxY8t3jgDgXGii7shMxlCvjcmFcEh2Xr+dVsD1GhSw3UBckKzv8W00z2nle8M9F+EkCzvQwviXVaiH2qCIlOs4v+p+WC0ERpBOtTqFtiF1quKDATU2mH+/5i3mm+jphXqaz5s6k9gf0Bmgx2La5cuIQXPDtWfMsqZKe7R3dlJwfL6qk5KrmcsUj2KZStlEZeFlCHTA1OHG7aY0dkeI3LpgNJkKKSfOgqEnznvXgqmQadMIWMoH7FyqCsPQNKbkUGAae0lzCMCXoB0mIf+MzIdm/F3Heaa0UqNzspjhZGaIDf/1vFZ4MCJtxU7K6CStDn6AJTlpRxD5XjFb7rdXOWwq6Bu0V/xS+Yuk615gud75l6vvhNw1j4mPOG+mcu07h4GI/y4G05YaeVhcVod8Y4uRBzQ==
EOF

cloud-localds -v ~/libvirt/seed.iso ~/libvirt/cidata/user-data ~/libvirt/cidata/meta-data

qemu-img create -f qcow2 -b /var/lib/libvirt/images/debian-9.4.3-20180416-openstack-amd64.qcow2 ~/libvirt/"$vm".qcow2 10g

virt-install --import \
        --name="$vm" \
        --ram=1024 \
        --vcpus=1 \
        --cpu host \
        --network bridge=br0 \
        --os-variant debian9 \
        --disk ~/libvirt/"$vm".qcow2 \
        --graphics spice \
        --disk ~/libvirt/seed.iso

virsh detach-disk --persistent "$vm" vdb

Debian doesn’t seem to like the graphics none for the install. Although, after starting the spice window, we can do virsh console asterisk-20180417220424-Debian-9.

Disable cloud-init

Sometimes cloud-init slows down boot. There’s little reason to run it after install. Disable cloud-init like:

#  touch /etc/cloud/cloud-init.disabled

Ansible for Post-Install Configuration

We use Ansible to do post-install configureation of the new instance.

$ ansible-playbook ~/ansible/lab-centos7.yaml --inventory 10.0.0.147,

…with the playbook like: