HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Nomad Home

Volume Specification

The Nomad volume specification defines the schema for creating and registering volumes using the volume create and volume register commands and the PUT /v1/volume/csi/:volume_id/create and PUT /v1/volume/csi/:volume_id API endpoints.

Some attributes are only be supported by specific operation, while others may have a different meaning for each action, so read the documentation for each attribute carefully. The section Differences Between Create and Register provides a summary of the differences.

The file may be provided as either HCL or JSON to the commands and as JSON to the API. An example HCL configuration for a volume create command:

id           = "ebs_prod_db1"
namespace    = "default"
name         = "database"
type         = "csi"
plugin_id    = "ebs-prod"
snapshot_id  = "snap-12345" # or clone_id, see below
capacity_max = "200G"
capacity_min = "100G"

capability {
  access_mode     = "single-node-reader-only"
  attachment_mode = "file-system"
}

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

mount_options {
  fs_type     = "ext4"
  mount_flags = ["noatime"]
}

topology_request {
  required {
    topology { segments { "rack" = "R2" } }
    topology { segments { "rack" = "R1", "zone" = "us-east-1a"} }
  }
  preferred {
    topology { segments { "rack" = "R1", "zone" = "us-east-1a"} }
  }
}

secrets {
  example_secret = "xyzzy"
}

parameters {
  skuname = "Premium_LRS"
}

Volume Specification Parameters

  • id (string: <required>) - The unique ID of the volume. This is how the volume.source field in a job specification will refer to the volume.

  • namespace (string: <optional>) - The namespace of the volume. This field overrides the namespace provided by the -namespace flag or NOMAD_NAMESPACE environment variable. Defaults to "default" if unset.

  • name (string: <required>) - The display name of the volume. On volume creation, this field may be used by the external storage provider to tag the volume.

  • type (string: <required>) - The type of volume. Currently only "csi" is supported.

  • external_id (string: <required>) - The ID of the physical volume from the storage provider. For example, the volume ID of an AWS EBS volume or Digital Ocean volume. Only allowed on volume registration.

  • plugin_id (string: <required>) - The ID of the CSI plugin that manages this volume.

  • snapshot_id (string: <optional>) - If the storage provider supports snapshots, the external ID of the snapshot to restore when creating this volume. If omitted, the volume will be created from scratch. The snapshot_id cannot be set if the clone_id field is set. Only allowed on volume creation.

  • clone_id (string: <optional>) - If the storage provider supports cloning, the external ID of the volume to clone when creating this volume. If omitted, the volume will be created from scratch. The clone_id cannot be set if the snapshot_id field is set. Only allowed on volume creation.

  • capacity_min (string: <optional>) - Option for requesting a minimum capacity, in bytes. The capacity of a volume may be the physical size of a disk, or a quota, depending on the storage provider. The specific size of the resulting volume will be somewhere between capacity_min and capacity_max; the exact behavior is up to the storage provider. If you want to specify an exact size, you should set capacity_min and capacity_max to the same value. Accepts human-friendly suffixes such as "100GiB". This field may not be supported by all storage providers. Increasing this value and re-issuing volume create or volume register may expand the volume, if the CSI plugin supports it.

  • capacity_max (string: <optional>) - Option for requesting a maximum capacity, in bytes. The capacity of a volume may be the physical size of a disk, or a quota, depending on the storage provider. The specific size of the resulting volume will be somewhere between capacity_min and capacity_max; the exact behavior is up to the storage provider. If you want to specify an exact size, you should set capacity_min and capacity_max to the same value. Accepts human-friendly suffixes such as "100GiB". This field may not be supported by all storage providers.

  • capability (Capability: <required>) - Option for validating the capability of a volume.

  • mount_options (MountOptions: <required>) - Options for mounting file-system volumes that don't already have a pre-formatted file system.

  • topology_request (TopologyRequest: nil) - Specify locations (region, zone, rack, etc.) where the provisioned volume must be accessible from in the case of volume creation or the locations where the existing volume is accessible from in the case of volume registration.

  • secrets (map<string|string>:nil) - An optional key-value map of strings used as credentials for publishing and unpublishing volumes.

  • parameters (map<string|string>:nil) - An optional key-value map of strings passed directly to the CSI plugin to configure the volume. The details of these parameters are specific to each storage provider, so consult the specific plugin documentation for more information.

  • context (map<string|string>:nil) - An optional key-value map of strings passed directly to the CSI plugin to validate the volume. The details of these parameters are specific to each storage provider, so consult the specific plugin documentation for more information. Only allowed on volume registration. Note that, like the rest of the volume specification, this block is declarative, and an update replaces it in its entirety, therefore all parameters need to be specified.

Differences Between Create and Register

Several fields are set automatically by the plugin when volume create or volume register commands are successful and you should not set their values if they are not supported by the operation.

You should not set the snapshot_id, or clone_id fields on volume registration.

And you should not set the external_id or context fields on volume creation.

Updating a Volume Definition

The volume register command allows updating a volume definition. But not all fields can be updated after the volume is registered:

  • The capacity_min and capacity_max fields can be updated, and may increase the volume size if the CSI plugin supports it. Expansion may or may not be possible while the volume is in use, again depending on the plugin. Reducing volume capacity is not allowed per the CSI spec.
  • The capability blocks can be added or removed, but only if the capability is not currently in use by a mounted volume.
  • The mount_options block can be updated if the volume is not in use.
  • The secrets block can be updated.
  • The context block can be updated. The values for this field are typically provided by the CSI plugin, and should not be updated unless recommended by the CSI plugin's documentation.

Volume Expansion

CSI Volumes may be expanded (increased in size) if the CSI controller plugin (and node plugin, if required) has the EXPAND_VOLUME capability.

To trigger a volume expansion, increase capacity_min above the current real capacity of the volume (as seen with the volume status command), and re-issue either volume create or volume register.

Nomad will reconcile the requested capacity by issuing expand volume requests to the controller plugin, and if required by the controller, also to the node plugins for each allocation that has a claim on the volume.

Examples

Volume registration

This is an example file used for the volume register command.

id              = "ebs_prod_db1"
name            = "database"
type            = "csi"
external_id     = "vol-23452345"
plugin_id       = "ebs-prod"

capability {
  access_mode     = "single-node-reader-only"
  attachment_mode = "file-system"
}

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

mount_options {
  fs_type     = "ext4"
  mount_flags = ["noatime"]
}

topology_request {
  required {
    topology { segments { "rack" = "R2" } }
    topology { segments { "rack" = "R1", "zone" = "us-east-1a"} }
  }
}

secrets {
  example_secret = "xyzzy"
}

parameters {
  skuname = "Premium_LRS"
}

context {
  endpoint = "http://192.168.1.101:9425"
}

Example Volume Expansion

Either volume create or volume register commands can trigger an expand to occur, after the volume has already been created or registered.

Example using volume create, with a volume.hcl file:

id        = "ebs_prod_db1"
name      = "database"
type      = "csi"
plugin_id = "ebs-prod"

capacity_min = "50gib"
capacity_max = "50gib"

# ... etc ...

Create the volume:

$ nomad volume create volume.hcl
Created external volume vol-asdf1234 with ID ebs_prod_db1

See its current capacity with volume status:

$ nomad volume status ebs_prod_db1 | grep Capacity
Capacity             = 50 GiB

Update volume.hcl:

id        = "ebs_prod_db1"
name      = "database"
type      = "csi"
plugin_id = "ebs-prod"
# all of the above must remain the same

capacity_min = "100gib" # double
capacity_max = "100gib" # increased to match

# ... etc ...

Run volume create again:

$ nomad volume create volume.hcl
Created external volume vol-asdf1234 with ID ebs_prod_db1

Check the volume capacity again:

$ nomad volume status ebs_prod_db1 | grep Capacity
Capacity             = 100 GiB

If you encounter errors that are not clear from the command output, the CSI plugin allocation logs and/or Nomad leader server logs may be helpful.

Edit this page on GitHub