How to Create a VM on Ubuntu with Terraform, Libvirt, and QEMU: Solving Real-World Issues

Creating virtual machines (VMs) on Linux using Terraform and Libvirt is an excellent way to automate infrastructure, but the process can come with its fair share of challenges. In this article, I’ll walk you through how to set up a VM on Ubuntu using Terraform, Libvirt, and QEMU, and explain all the real-world issues I faced and solved along the way.

How to Create a VM on Ubuntu with Terraform, Libvirt, and QEMU: Solving Real-World Issues



Table of Contents
  • Introduction
  • Prerequisites
  • Installing Required Packages
  • Setting Up the Libvirt Pool
  • Preparing the QCOW2 Image
  • Writing the Terraform Configuration
  • Common Errors and How to Fix Them
  • Managing AppArmor
  • Final Working Script

Introduction

Virtualization is a key component of modern infrastructure. With tools like Libvirt and QEMU, you can run lightweight VMs on a local or remote Linux host. Terraform adds the power of Infrastructure as Code to define and deploy VMs automatically.

But things rarely work the first time.

Prerequisites

  • Ubuntu Linux (20.04 or later)

  • A user with sudo privileges

  • Terraform installed (sudo apt install terraform or from Terraform's website)

  • Libvirt and QEMU installed

sudo apt update

sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst libvirt-dev


  • Enable and start the libvirtd service:

sudo systemctl enable --now libvirtd

Installing Required Packages

In this step, we install Terraform and configure it to work with the Libvirt provider, which is not officially maintained by HashiCorp but provided by the community. Terraform does not include the Libvirt provider by default, so we must set up the HashiCorp apt repository and install the Terraform CLI. After this, we will be ready to configure and apply infrastructure plans using the Libvirt API.

Install the Terraform Libvirt provider:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install terraform

Setting Up the Libvirt Pool

A Libvirt storage pool is a logical unit that defines where virtual machine images and volumes are stored. Think of it as a container or directory that Libvirt uses to manage disk resources. Pools can point to local directories, LVM volumes, NFS mounts, and more. In our case, we're using a simple directory-based pool located at /var/lib/libvirt/images/terraform. This pool must be readable and writable by the libvirt-qemu user, which is the default user running QEMU processes.

You can use the default pool (/var/lib/libvirt/images) or create a custom directory, e.g., /var/lib/libvirt/images/terraform. Make sure it’s accessible:

sudo mkdir -p /var/lib/libvirt/images/terraform

sudo chown libvirt-qemu:libvirt-qemu /var/lib/libvirt/images/terraform

sudo chmod 755 /var/lib/libvirt/images/terraform



Setting Up the Libvirt Pool


Preparing the QCOW2 Image

In this step, we acquire and prepare the virtual disk image that will serve as the hard drive for our virtual machine. We are using a QCOW2 image, which is a common format for QEMU virtual disks. Specifically, we download a Ubuntu cloud image, which is optimized for cloud environments and typically used with cloud-init. Once downloaded, we move the image to the storage pool directory managed by Libvirt, ensure the proper ownership (libvirt-qemu) and permissions (644), and verify the integrity and format of the image using qemu-img info.

Download a valid cloud image:

wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img -O ubuntu-vm.qcow2

sudo cp ubuntu-vm.qcow2 /var/lib/libvirt/images/terraform/

sudo chown libvirt-qemu:libvirt-qemu /var/lib/libvirt/images/terraform/ubuntu-vm.qcow2

sudo chmod 644 /var/lib/libvirt/images/terraform/ubuntu-vm.qcow2

Check the image:

qemu-img info /var/lib/libvirt/images/terraform/ubuntu-vm.qcow2


Writing the Terraform Configuration

In this section, we define the infrastructure using Terraform's HCL (HashiCorp Configuration Language). Here's a breakdown of the file:

  • The terraform block specifies which providers are needed and from where to download them. In our case, we use dmacvicar/libvirt to interact with the Libvirt API.

  • The provider "libvirt" block defines how Terraform connects to Libvirt, using the system URI.

  • The libvirt_pool resource defines a storage pool—a directory where disk images will be stored. We use /var/lib/libvirt/images/terraform.

  • The libvirt_volume resource describes the disk image used for the VM. It points to the previously downloaded .qcow2 file.

  • The libvirt_domain resource defines the actual virtual machine:

    • name: Name of the VM.

    • memory, vcpu: Resources allocated.

    • disk: Connects the previously defined volume.

    • network_interface: Connects the VM to a default virtual network.

    • console: Enables serial access for debugging or login.

    • graphics: Enables VNC access to view the VM graphically.

Below is a complete working example:

Here is a basic working example of main.tf:


terraform {

  required_providers {

    libvirt = {

      source  = "dmacvicar/libvirt"

      version = "0.6.14" # o la versione che hai installato

    }

  }

}


provider "libvirt" {

  uri = "qemu:///system"

}


resource "libvirt_pool" "user_pool" {

  name = "terraform-pool"

  type = "dir"

  path = "/var/lib/libvirt/images/terraform"

}


resource "libvirt_volume" "ubuntu" {

  name   = "ubuntu-vm.qcow2"

  pool   = libvirt_pool.user_pool.name

  source = "/var/lib/libvirt/images/terraform/ubuntu-vm.qcow2"

  format = "qcow2"

}


resource "libvirt_domain" "ubuntu_vm" {

  name   = "ubuntu-vm"

  memory = 2048

  vcpu   = 2


  disk {

    volume_id = libvirt_volume.ubuntu.id

  }


  network_interface {

    network_name = "default"

  }


  console {

    type        = "pty"

    target_type = "serial"

    target_port = "0"

  }


  graphics {

    type = "vnc"

    listen_type = "address"

  }

}


Common Errors and How to Fix Them

Permission Denied on .qcow2 File

  • Ensure the image is readable by libvirt-qemu

  • Check directory traversal permissions:

     sudo chmod o+x /var /var/lib /var/lib/libvirt /var/lib/libvirt/images /var/lib/libvirt/images/terraform

qemu-system-x86_64: Could not open file

  • File may be corrupted or incomplete. Re-download.

  • Check with qemu-img info

Terraform fails to delete pool: Directory not empty

  • Avoid using /var/lib/libvirt/images directly. Use a subdirectory.

  • Add prevent_destroy = true if needed.

Managing AppArmor

AppArmor (Application Armor) is a Linux security module that restricts programs' capabilities with per-application profiles. It works by enforcing a set of rules that define what files or system resources a specific program can access. These profiles are loaded into memory at boot or when the corresponding applications start, and they can either enforce access restrictions (enforce mode) or simply log violations without blocking them (complain mode).

While AppArmor adds an important security layer to your system, it can also silently interfere with virtual machine creation—especially when using non-standard paths or custom images. This was the trickiest part of the VM setup process. Even with all permissions correct, AppArmor blocked QEMU from accessing the image file:


DENIED operation="open" profile="libvirt-..." name="...ubuntu-vm.qcow2" ...


Fix Options:

Temporarily Disable AppArmor usr.sbin.libvirtd and usr.bin.qemu-system-x86_64 profiles

sudo apparmor_parser -R "/etc/apparmor.d/usr.sbin.libvirtd"

sudo apparmor_parser -R "/etc/apparmor.d/usr.bin.qemu-system-x86_64"

sudo systemctl restart libvirtd


These commands remove the AppArmor profiles from memory temporarily. They do not delete the profiles from disk.

Profiles will be restored automatically at the next reboot or when you run:

sudo systemctl restart apparmor

or

sudo apparmor_parser /etc/apparmor.d/usr.sbin.libvirtd

sudo apparmor_parser /etc/apparmor.d/usr.bin.qemu-system-x86_64


Final Working Script

This script disables the AppArmor profiles that interfere with QEMU and Libvirt, restarts the libvirt daemon to apply changes, and then runs terraform apply to create the virtual machine. It is useful when AppArmor is blocking access to image files despite correct Unix permissions. The profiles will be automatically restored on the next reboot or if AppArmor is restarted manually.


#!/bin/bash


sudo apparmor_parser -R "/etc/apparmor.d/usr.sbin.libvirtd"

sudo apparmor_parser -R "/etc/apparmor.d/usr.bin.qemu-system-x86_64"


sudo systemctl restart libvirtd


terraform apply


Conclusion

Creating VMs with Terraform and Libvirt on Ubuntu is powerful, but not without its hurdles. The biggest gotcha is AppArmor, which silently blocks access even when file permissions are perfect. With the right configuration, clear understanding of permissioning, and attention to security profiles like AppArmor, you can deploy VMs reliably and automatically.

References
Location: Roma RM, Italia

Comments

Post a Comment

Popular posts from this blog

Monitoring and Logging with Prometheus: A Practical Guide

Image
Monitoring and logging are critical components of any robust IT infrastructure. They ensure that systems run smoothly, issues are detected early, and performance is optimized. Prometheus has emerged as a popular tool for these tasks, offering powerful features for metrics collection, querying, and alerting. In this guide, we'll explore how to effectively use Prometheus for monitoring and logging, providing a practical, hands-on approach to get you started. Key topics include Prometheus architecture, installation, setup, metric collection, alerting, and integration with other tools. If you find yourself curious about who is utilizing Prometheus globally, here is a brief list of some notable companies that have adopted Prometheus for their needs: JPMorgan, Dice, Garfana Labs, Visa,  more ... What is Prometheus? Understanding the Basics Prometheus is an open-source monitoring and alerting toolkit designed specifically for reliability and scalability. Originally developed at SoundCloud...
Read more

Creating a Complete CRUD API with Spring Boot 3: Authentication, Authorization, and Testing

Image
In this article, I will explore how to create a complete application that implements a CRUD API using Spring Boot 3. I will perform Create, Read, Update, and Delete operations on a database, implement authentication and authorization with JSON Web Token (JWT), and develop a REST API for managing orders in an e-commerce system. Finally, I will test the API using JUnit. Table of Contents Technologies Used Client-Server Interactions for Order Retrieval Creating a CRUD API Project Setup Structuring the Project Database Configuration Creating Entity and Repository Implementing Service and Controller Authentication and Authorization with JWT Configuring Spring Security Creating JWT Tokens Implementing the Login Endpoint Swagger setup Testing APIs with JUnit Testing the Service Testing the Controller Running Tests with Maven Testing Authentication Testing Order Retrieval with Authentication Advanced Topics and Best Practices Conclusion References Te...
Read more

Logging in Spring Boot 3: Best Practices for Configuration and Implementation

Image
Logging in a Spring Boot 3 application is essential for monitoring system behavior, diagnosing issues, and analyzing performance. Well-structured logging helps maintain a clear view of the application flow, identify anomalies, and optimize debugging. However, improper use of logging can negatively impact application performance and security. In this article, we will explore best practices for configuring and writing logs effectively, ensuring scalability, readability, and compliance with data protection regulations. Installing Lombok for Logging Lombok is a library that simplifies boilerplate code management in Java, including logger declarations. If you want to use @Slf4j to handle logs in a cleaner way, follow these steps to install Lombok. Adding Lombok as a Dependency If you use Maven, add this dependency to your pom.xml:. <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <scope>provided</scope> </depende...
Read more