Parallel builds

5min
|
Packer

In the previous tutorials, you used Packer to automatically build an image and provision it from a template. Then, you parameterized the template allowing you to build different images using the same template. While this is already quite powerful, Packer can create multiple images in parallel, all configured from a single template.

Parellel build is a very useful and important feature of Packer. For example, Packer can build an Amazon AMI and a VMware virtual machine in parallel provisioned with the same scripts, resulting in near-identical images. The AMI can be used for production, the VMware machine can be used for development. Or, another example, if you're using Packer to build software appliances, then you can build the appliance for every supported platform all in parallel and configured from a single template.

In this tutorial, you will build a second Docker image in parallel. While the second image is a Docker image for the sake of this tutorial, the second source could just as easily come from another hypervisor or a cloud platform.

Prerequisites

This tutorial assumes that you are continuing from the previous tutorials. If not, follow the steps below before continuing.

Install Packer

Create a directory named packer_tutorial and paste the following configuration into a file named docker-ubuntu.pkr.hcl.

packer {
  required_plugins {
    docker = {
      version = ">= 1.0.8"
      source  = "github.com/hashicorp/docker"
    }
  }
}

variable "docker_image" {
  type    = string
  default = "ubuntu:jammy"
}

source "docker" "ubuntu" {
  image  = var.docker_image
  commit = true
}

build {
  name    = "learn-packer"
  sources = [
    "source.docker.ubuntu",
  ]

  provisioner "shell" {
    environment_vars = [
      "FOO=hello world",
    ]
    inline = [
      "echo Adding file to Docker Container",
      "echo \"FOO is $FOO\" > example.txt",
    ]
  }

  provisioner "shell" {
    inline = ["echo Running ${var.docker_image} Docker image."]
  }
}

Initialize the Packer template.
```
$ packer init .
```

Once you have successfully initialized the template, you can continue with the rest of this tutorial.

Add parallel builds to template

To use parallel builds, create a source then add the source to the sources array in the build block. Your sources do not need to be the same type. This tells Packer to build multiple images when that build is run.

Add the following source block to your docker-ubuntu.pkr.hcl file. Do not replace the existing ubuntu source; add it underneath.

source "docker" "ubuntu-focal" {
  image  = "ubuntu:focal"
  commit = true
}

Then update your build block to use the new source.

build {
  name    = "learn-packer"
  sources = [
-    "source.docker.ubuntu"
+    "source.docker.ubuntu",
+    "source.docker.ubuntu-focal",
  ]
  ## ...
}

Finally, let's modify our shell provisioner to output the OS of each build instead of our docker_image variable which is static:

provisioner "shell" {
    inline = ["echo Running $(cat /etc/os-release | grep VERSION= | sed 's/\"//g' | sed 's/VERSION=//g') Docker image."]
  }

Build images

Build the images. Packer will display color-coded output for both builds. You can tell which build source an output line is associated with using either the line color or prefix. In this example, there are two builds: docker.ubuntu and docker.ubuntu-focal.

$ packer build .
learn-packer.docker.ubuntu: output will be in this color.
learn-packer.docker.ubuntu-focal: output will be in this color.

==> learn-packer.docker.ubuntu: Creating a temporary directory for sharing data...
==> learn-packer.docker.ubuntu-focal: Creating a temporary directory for sharing data...
==> learn-packer.docker.ubuntu: Pulling Docker image: ubuntu:focal
==> learn-packer.docker.ubuntu-focal: Pulling Docker image: ubuntu:focal
    learn-packer.docker.ubuntu: focal: Pulling from library/ubuntu
    learn-packer.docker.ubuntu: Digest: sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8
    learn-packer.docker.ubuntu: Status: Image is up to date for ubuntu:focal
    learn-packer.docker.ubuntu: docker.io/library/ubuntu:focal
==> learn-packer.docker.ubuntu: Starting docker container...
    learn-packer.docker.ubuntu: Run command: docker run -v /Users/youruser/.packer.d/tmp396782610:/packer-files -d -i -t --entrypoint=/bin/sh -- ubuntu:focal
    learn-packer.docker.ubuntu-focal: focal: Pulling from library/ubuntu
    learn-packer.docker.ubuntu-focal: Digest: sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8
    learn-packer.docker.ubuntu-focal: Status: Image is up to date for ubuntu:focal
    learn-packer.docker.ubuntu-focal: docker.io/library/ubuntu:focal
==> learn-packer.docker.ubuntu-focal: Starting docker container...
    learn-packer.docker.ubuntu-focal: Run command: docker run -v /Users/youruser/.packer.d/tmp049683550:/packer-files -d -i -t --entrypoint=/bin/sh -- ubuntu:focal
    learn-packer.docker.ubuntu: Container ID: 32f3b56578ebadb6a180ed8bfcc819d5282e2c7c680719846d46b15c961cb700
==> learn-packer.docker.ubuntu: Using docker communicator to connect: 172.17.0.3
    learn-packer.docker.ubuntu-focal: Container ID: 44023b88cc7fa1389fd776335a1971caf8598ab0493846ea158c3a1eb743c537
==> learn-packer.docker.ubuntu-focal: Using docker communicator to connect: 172.17.0.4
==> learn-packer.docker.ubuntu: Provisioning with shell script: /var/folders/s6/m22_k3p11z104k2vx1jkqr2c0000gp/T/packer-shell254926715
==> learn-packer.docker.ubuntu-focal: Provisioning with shell script: /var/folders/s6/m22_k3p11z104k2vx1jkqr2c0000gp/T/packer-shell790094151
    learn-packer.docker.ubuntu: Adding file to Docker Container
    learn-packer.docker.ubuntu-focal: Adding file to Docker Container
==> learn-packer.docker.ubuntu: Provisioning with shell script: /var/folders/s6/m22_k3p11z104k2vx1jkqr2c0000gp/T/packer-shell076100704
==> learn-packer.docker.ubuntu-focal: Provisioning with shell script: /var/folders/s6/m22_k3p11z104k2vx1jkqr2c0000gp/T/packer-shell915744804
    learn-packer.docker.ubuntu: Running ubuntu:focal Docker image.
    learn-packer.docker.ubuntu-focal: Running ubuntu:focal Docker image.
==> learn-packer.docker.ubuntu: Committing the container
==> learn-packer.docker.ubuntu-focal: Committing the container
    learn-packer.docker.ubuntu: Image ID: sha256:addef89c2f2af5ecf88de93c5e7fe578c673281cd78dfbd1f9be6cd3690a2423
    learn-packer.docker.ubuntu-focal: Image ID: sha256:12a067ffaae0bd1c5a7957c4f974565a79e76a3bada57815ffc8ab544fd445b0
==> learn-packer.docker.ubuntu: Killing the container: 32f3b56578ebadb6a180ed8bfcc819d5282e2c7c680719846d46b15c961cb700
==> learn-packer.docker.ubuntu-focal: Killing the container: 44023b88cc7fa1389fd776335a1971caf8598ab0493846ea158c3a1eb743c537
Build 'learn-packer.docker.ubuntu' finished after 28 seconds 742 milliseconds.
Build 'learn-packer.docker.ubuntu-focal' finished after 28 seconds 813 milliseconds.

==> Wait completed after 28 seconds 813 milliseconds

==> Builds finished. The artifacts of successful builds are:
--> learn-packer.docker.ubuntu: Imported Docker image: sha256:addef89c2f2af5ecf88de93c5e7fe578c673281cd78dfbd1f9be6cd3690a2423
--> learn-packer.docker.ubuntu-focal: Imported Docker image: sha256:12a067ffaae0bd1c5a7957c4f974565a79e76a3bada57815ffc8ab544fd445b0

Next steps

In this tutorial, you updated your template to build multiple images in parallel. Continue to the next tutorial to add a post-processing step to your Packer template.

Refer to the following resources for additional details on the concepts covered in this tutorial:

Learn how to restrict provisioners to certain build sources with the only and except attributes.

Variables

Post-processors