Users of cloud computing platforms like AWS, Azure, and OpenStack want flexibility in their instance types and to customize their instances. Users minimally need to inject SSH keys into new instances, but also want to set hostnames, timezone, etc., and even run custom scripts. OS vendors don’t want to master different qcow2 images for different cloud vendors, and they definitely don’t want different images for each instance type. Cloud providers, like Amazon, want to meet customer needs, while keeping the whole thing as simple as possible and automated.

Cloud-init solves the problem. Different cloud providers can use the same image to automatically provision various machine types, and users can set many options before spinning up new instances.

Where does cloud-init get its customization data? How does it know where to look? The default configuration file for cloud-init is /etc/cloud/cloud.cfg. The cloud.cfg, like cloud-config, uses YAML format. It can contain a datasource section:

# Example datasource config
# datasource: 
#    Ec2: 
#      metadata_urls: [ '' ]
#      timeout: 5 # (defaults to 50 seconds)
#      max_wait: 10 # (defaults to 120 seconds)

But a big part of cloud-init is using stock images, where we won’t have the opportunity to customize cloud.cfg prior to the first boot. So, how does cloud-init know where to find its metadata? To what values does datasource default? Cloud-init has a built-in list of supported datasources. During early boot, before the network comes up, cloud-init looks for local data sources. The details vary depending on the provider, but often it’s either:

In what order does it check the dozen+ potential datasource providers? I don’t know, but it seems to check the local ones before the network ones.

Apart from /etc/cloud/, cloud-init stores files in /var/lib/cloud/.


The cloud-config syntax uses YAML.

On AWS, using the instance creation wizard, we add cloud-config data in the “User Data” field of the “Instance Details” step.

The official cloud-init docs offer many examples.

Cloud-config example of adding a user:

	- name: paul
		groups: sudo
		shell: /bin/bash
		sudo: ['ALL=(ALL) NOPASSWD:ALL']
			ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf0q4P7G0doiBQYV7OlOxbRjle026hJPBWDAaaaaxcdxd

Cloud-config example of adding a repo:

        enabled: false
        failovermethod: priority
        gpgcheck: true
        gpgkey: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
        name: Extra Packages for Enterprise Linux 7 - Testing

Cloud-config example of updating a Debian-flavored instance and installing packages:

apt_update: true
apt_upgrade: true
	- apache2
	- rsync

Cloud-config example of running arbitrary commands:

	- [ wget, "", -O, /tmp/index.html ]
	- /bin/echo 'Hello, world!' > /var/www/index.html ]

Cloud-init for a virtual test lab (KVM/libvirt)

Use KVM, libvirt, QEMU, and cloud-init to spin up instances for testing and experimentation. Create each instance from a qcow2 base image supplied by the OS distro, then customize the instances with cloud-init.

The NoCloud datasource accepts data from an attached disk image. The libguestfs-tools package provides the tool virt-make-fs, which helps create the “cidata” disk image.

Cloud-init expects two files on the NoCloud “cidata” disk image: a meta-data file and a user-data file. The idea behind the two files is for the cloud provider to supply the metadata, and the customer/user to supply the user data. Since we’re acting as our own cloud provider, we supply both.

The meta-data file looks like:

instance-id: iid-012345
local-hostname: myvm-CentOS-7-x86_64

Our YAML cloud-config user-data file looks something like:

timezone: America/Detroit
locale: en_US.UTF-8
  - name: paul
    groups: sudo
    shell: /bin/bash
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqia4sRZ+v6Xuvv4HrTf8R4wIHAilul7LIlR5LTKzHvaCXrTAVkIqUhK4cJMEXCOKyS7kOnAkwNzKj/NPQNfw42JQznGF3I6VT8HTqi2NfTQ4lqe+vaMiCZU9mW8q33J3SJzLD/UciA2SqqSr3QMI9urDLn3ovpyKqYjxnSo1f9i+LInmXGgF7muWbGAXpWJyiUJxNB0WzZa3NpCmexlXw6f27tR1+fSJUDDP/HfwimmZms7Q8hT03dohI5OJmbI2YAX+OrHfO4Bya1jeQ5vLKoEX7jywLXHhuJORJ+6fUZBGL6sMQGfWKbYThdpU8MDmNSlOdHl9inyoSi69QZ2bJ

Stick the two files in a directory, and make the disk image:

$ mkdir -p ~/libvirt/cidata
$ cp meta-data ~/libvirt/cidata/
$ cp user-data ~/libvirt/cidata/
$ virt-make-fs --type=vfat --label=cidata ~/libvirt/cidata ~/libvirt/cidata.vfat
$ virt-filesystems -a ~/libvirt/cidata.vfat --all --long -h

Here’s how we create an instance from a base qcow2 image, customized with cloud-init:

$ wget
$ tar xf CentOS-7-x86_64-GenericCloud-1802.qcow2.xz
# mv CentOS-7-x86_64-GenericCloud-1802.qcow2 /var/lib/libvirt/images/
$ mkdir -p ~/libvirt
$ qemu-img create -f qcow2 -b /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud-1802.qcow2 ~/libvirt/myvm-CentOS-7-x86_64.qcow2 10G
$ virt-install --import \
	--name=myvm-CentOS-7-x86_64 \
	--ram=1024 \
	--vcpus=1 \
	--network bridge=br0 \
	--os-variant rhel7.4 \
	--graphics none \
	--disk ~/libvirt/myvm-CentOS-7-x86_64.qcow2 \
	--disk ~/libvirt/cidata.vfat

A different way? NoCloud also looks for meta-data in the SMBIOS “serial” string, like:


virt-install accepts SMBIOS settings through its --sysinfo option.

Presumably, serving a meta-data and user-data file from “” (or wherever) would then be sufficient.

$ virt-install --sysinfo type=smbios,system_serial=ds=nocloud-net;s= [...]

…or maybe:

$ virt-install --qemu-commandline="-smbios type=1,serial=ds=nocloud-net;s=" [...]

I have not yet gotten this to work.

virt-install graphics

Changing the virt-install graphics from default to --graphics none shows much more cloud-init debug information. I don’t know why output to vnc or qxl graphics seems to end before showing the cloud-init information.

Hmm. The console output make it clear that “nocloud-net” isn’t simply trying to grab the meta-data and user-data files from the URL. It’s trying to access values at some some sort of tree-like key-value store.

[  141.714579] cloud-init[849]: 2018-04-02 10:43:10,403 -[WARNING]: Calling '' failed [0/120s]: bad status code [404]
[  142.724408] cloud-init[849]: 2018-04-02 10:43:11,412 -[WARNING]: Calling '' failed [1/120s]: bad status code [404]
[  261.103099] cloud-init[849]: 2018-04-02 10:45:09,790 -[WARNING]: Calling '' failed [119/120s]: request error [('Connection aborted.', error(115, 'Operation now in progress'))]
[  268.120181] cloud-init[849]: 2018-04-02 10:45:16,798 -[CRITICAL]: Giving up on waiting for the metadata from [''] after 126 seconds

It’s probably simpler after all to just attach a vfat drive with the meta-data and user-data files.