Parameterized jobs on Nomad

19min
|
Nomad

Nomad parameterized jobs enable dynamic runtime behaviors from a single job specification. These specialized batch-type jobs act less like regular Nomad jobs and more like functions. They are reusable, can accept parameters, and run only when invoked.

Parameterized jobs provide three key benefits to end users.

Creating instances of the job without resubmitting the job specification.
Providing dynamic run-time values as parameters to the job during dispatch.
Proving, when the job allows, an opaque payload that Nomad will write to one or more of the job's tasks.

Nomad registers, but does not run, a parameterized job when you submit it to the cluster. To run a parameterized job, end-users invoke—or dispatch—it with the nomad job dispatch command or Nomad Job Dispatch API.

During dispatch, the user provides the parameter values and payload. Nomad combines these values with the registered job, creates a dispatched instance, and returns the scheduling information to the end-user. Once dispatched, the created instances are handled like regular Nomad batch jobs.

Challenge

In this tutorial, you will learn the fundamentals of Nomad parameterized jobs. You will take a Nomad batch job that renders a template, and

Convert it to a parameterized job
Enhance it with optional and required parameters
Define default values for optional parameters
Handle dispatch payloads

These fundamental practices can be used to create more complex parameterized workloads in your own environment.

Prerequisites

Nomad v1.0 or greater
Nomad dev agent or Nomad cluster

Configure your learning environment

Fetch the tutorial content

This tutorial uses content provided in the hashicorp/learn-nomad-jobspec repository on GitHub. You can download a ZIP archive directly or use git to clone the repository.

$ wget https://github.com/hashicorp/learn-nomad-jobspec/archive/release.zip

Unarchive the downloaded release.

$ unzip release.zip

The unzipping process creates a directory named learn-nomad-jobspec-release. Change into it and into the tutorial's folder.

$ cd learn-nomad-jobspec-release/parameterized

Clone the hashicorp/learn-nomad-jobspec repository.

$ git clone https://github.com/hashicorp/learn-nomad-jobspec.git

Change into the project directory.

$ cd learn-nomad-jobspec

Check out the release tag.

$ git checkout release

Change to the tutorial's folder

$ cd parameterized

Configure tutorial environment

This tutorial uses HCL2 variables to reduce the amount of editing that you need to do to the job specification based on your environment. The defaults are optimized for a Linux/macOS boxes targeting a local Nomad dev agent using the default datacenter value of dc1. These values can be customized using environment variables before running the job file.

Override default task driver (optional)

This tutorial uses either the raw_exec or exec task driver (raw_exec by default) to render a text file and output it to standard output.

Because Windows and macOS do not support the exec driver and the raw_exec driver is not enabled by default on non-dev agents, this job uses an HCL2 variable to let you set your preferred driver without editing the job.

The driver variable defaults to raw_exec, suitable for all learners using a dev agent. If you need to use the exec driver instead, set the NOMAD_VAR_driver environment variable to exec.

$ export NOMAD_VAR_driver=exec

Override default datacenter (optional)

The job also allows you to specify your datacenter value if it differs from the default value of dc1. If you are using a non-default datacenter value, set the NOMAD_VAR_datacenter environment variable.

If you are unsure of a valid datacenter value for your target cluster or dev agent, run the nomad node status command to list all the datacenters available.

$ nomad node status
ID        DC    Name                            Class   Drain  Eligibility  Status
d715f8b4  mydc  nomad-client-1.node.consul      <none>  false  eligible     ready
14ab9290  mydc  nomad-client-2.node.consul      <none>  false  eligible     ready
0f357b26  mydc  nomad-client-3.node.consul      <none>  false  eligible     ready

In this sample output, all the clients are in a datacenter named mydc, based on this output you would run:

$ export NOMAD_VAR_datacenter=mydc

Run the starting job

To get you started, the tutorial repository includes a Nomad batch job specification that you will use as a base to build on. Use the nomad job run command to run the start.nomad job file.

$ nomad job run start.nomad
==> Monitoring evaluation "68b8b7dc"
    Evaluation triggered by job "template"
    Allocation "cb1d80a1" created: node "66d36f06", group "renderer"
==> Monitoring evaluation "68b8b7dc"
    Allocation "cb1d80a1" status changed: "pending" -> "complete" (All tasks have completed)
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "68b8b7dc" finished with status "complete"

Make note of the allocation ID shown in your output. In the provided sample output, the allocation ID is cb1d80a1.

Fetch the output of the job

Since this job writes its output to standard output, you can retrieve it with the nomad alloc logs command. Run the nomad alloc logs command, supplying the allocation ID you noted in the previous step.

Pro Tip - You only need to provide enough of the allocation ID to The nomad alloc logs command to point to a distinct allocation. This works with many identifiers in the Nomad command-line tool.

$ nomad alloc logs cb1d80a1
Hello,
Thanks for your interest. I have attached the information
you requested. I look forward to talking with you soon.

Purge the job

Nomad will not allow you to convert a batch job to a parameterized job without purging the original version as a safety feature. When run with the -purge flag, the nomad job stop command deletes the job and information about any former runs of it from the Nomad server state.

Note

The start.nomad file defines a Nomad job named template. The filename and job name are intentionally different allowing you to keep copies of the different states of the job as you progress through the tutorial.

$ nomad job stop -purge template
==> Monitoring evaluation "a0704fca"
    Evaluation triggered by job "template"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "a0704fca" finished with status "complete"

Make the job parameterized

Adding a parameterized stanza converts a batch-type job to a parameterized one. An empty parameterized stanza creates a parameterized job that isn't customizable at dispatch time. This is still a useful feature, since it enables an operator or application to run an instance of the job without having to have the job specification itself.

Add parameterized stanza

Copy the start.nomad file to a new file named parameterized.nomad.

$ cp start.nomad parameterized.nomad

Open parameterized.nomad in a text editor. Add an empty parameterized stanza inside of the job stanza and save the file.

  parameterized {

  }

Register the job

Run the parameterized version of the job.

$ nomad job run parameterized.nomad
Job registration successful

Notice that the output doesn't show any scheduling activity—no evaluation or allocation information. Parameterized jobs themselves do not run.

If you get an error

If you receive the following error, it indicates that you missed purging the non-parameterized version of the template job. Run nomad job stop -purge template and re-run nomad job run parameterized.nomad to resolve it.

$ nomad job run template.nomad
Error submitting job: Unexpected response code: 500 (cannot update non-parameterized job to being parameterized)

Run the nomad job status command to verify your parameterized job is available for dispatch.

$ nomad job status
ID        Type                 Priority  Status   Submit Date
template  batch/parameterized  50        running  2021-04-11T22:01:45-04:00

Dispatch the job

Run the nomad job dispatch command to dispatch an instance of the parameterized job. The nomad job dispatch command's final argument is the registered job ID to dispatch, unlike the nomad job run command which uses the filename of the job specification.

$ nomad job dispatch template
Dispatched Job ID = template/dispatch-1620682960-e9dfcaf8
Evaluation ID     = 897c8095

==> Monitoring evaluation "897c8095"
    Evaluation triggered by job "template/dispatch-1620682960-e9dfcaf8"
    Allocation "4e317cb8" created: node "66d36f06", group "renderer"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "897c8095" finished with status "complete"

Examine the output of the command. There are some key differences from the typical output of the nomad job run command. Notice that Nomad generates a Dispatched Job ID. This generated job ID refers to this specific instance of the parameterized job, and it enables you to manage discrete instances of a parameterized job in the same ways you manage your other Nomad batch jobs.

The output also provides scheduling information. Collect the allocation ID from your output. In the preceding output it's 4e317cb8. Like earlier, run the nomad alloc logs command for your allocation ID.

$ nomad alloc logs 4e317cb8
Hello,
Thanks for your interest. I have attached the information
you requested. I look forward to talking with you soon.

Parameterized jobs without variables can be used to provide a means for running a batch job without having to supply the job specification.

Use dispatch variables

Parameterized jobs also provide the ability to submit run-specific variables while dispatching the job. The job specification defines the variable names. They can be optional or required.

Nomad provides the dispatched variable values as environment variables. The environment variable names that Nomad creates for these variables are prepended with NOMAD_META_, for example NOMAD_META_customer_email.

Nomad will not create environment variables for unset optional variables; your workload should handle this case. You may also define defaults for optional variables as part of the job specification.

Update the job to use email.tmpl

Copy the parameterized.nomad file to a new file named variables.nomad.

$ cp parameterized.nomad variables.nomad

Open the variables.nomad file in a text editor and change the HCL2 file function to read in the email.tmpl file. This template generates an email-like output.

Change

${file("./templates/template.tmpl")}

${file("./templates/email.tmpl")}

Define job variables

Required and optional variables are defined in the meta_required and meta_optional attributes of the parameterized stanza, respectively. These attributes take a list of quoted variable names the value.

The email.tmpl template needs more information to render properly:

required - customer_email, customer_name
optional - rep_name, rep_email, rep_title, product_name

Define the variables by adding these attributes to the variables.nomad file inside of the parameterized stanza.

    meta_required = ["customer_name","customer_email"]
    meta_optional = ["rep_name","rep_email","rep_title","product_name"]

Provide defaults for optional values

You can define default values for your optional variables using a job-level meta stanza. Use the variable name as the attribute name and set the value to the default.

For this example, create a default for the rep_name and the rep_email values to provide generic values when the rep_name and rep_email are not provided.

In the variables.nomad file, add this meta stanza inside of the job stanza.

  meta {
    rep_name = "BabbageCorp"
    rep_email = "hello@mechanicalcomputing.com"
  }

Deploy and dispatch the job

Save the variables.nomad file and submit it using the nomad job run command.

$ nomad job run variables.nomad
Job registration successful

Dispatch the job with variables

Now, dispatch a copy with the nomad job dispatch command. Use the -meta flag to set the customer_email to alovelace@programmers.net and customer_name to Ada Lovelace.

$ nomad job dispatch \
  -meta customer_email="alovelace@programmers.net" \
  -meta customer_name="Ada Lovelace" \
  template

The command will output scheduling information.

Dispatched Job ID = template/dispatch-1620750077-463bca71
Evaluation ID     = 1b407851

==> Monitoring evaluation "1b407851"
    Evaluation triggered by job "template/dispatch-1620750077-463bca71"
==> Monitoring evaluation "1b407851"
    Allocation "391fc0c0" created: node "f7bc1f2d", group "renderer"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "1b407851" finished with status "complete"

Fetch the template output using the nomad alloc logs command for the allocation ID output when you dispatched the job.

$ nomad alloc logs 391fc0c0
TO: Ada Lovelace <alovelace@programmers.net>
FROM: BabbageCorp <hello@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest. I look forward to talking with you soon.

Sincerely,
BabbageCorp

Test the required variables

Nomad validates that the dispatch request contains all required variables. The operator or application dispatching the job must provide all required variables when dispatching the job. If not, Nomad will return an error.

Observe this by dispatching the job with no variables set.

$ nomad job dispatch template
Failed to dispatch job: Unexpected response code: 500 (rpc error: Dispatch did not provide required meta keys: [customer_email customer_name])

Use the optional variables

Now, use the optional variables to personalize the email. Dispatch the job again with the following command.

$ nomad job dispatch \
  -meta customer_email="alovelace@programmers.net" \
  -meta customer_name="Ada Lovelace" \
  -meta rep_name="Charles Babbage" \
  -meta rep_email="cbabbage@mechanicalcomputing.com" \
  -meta rep_title="Chief Innovation Officer" \
  -meta product_name="Analytical Engine" \
  template

The command outputs the scheduling information.

Dispatched Job ID = template/dispatch-1620750369-9431cc3b
Evaluation ID     = 6312c1e3

==> Monitoring evaluation "6312c1e3"
    Evaluation triggered by job "template/dispatch-1620750369-9431cc3b"
    Allocation "902a3d43" created: node "f7bc1f2d", group "renderer"
==> Monitoring evaluation "6312c1e3"
    Allocation "902a3d43" status changed: "pending" -> "complete" (All tasks have completed)
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6312c1e3" finished with status "complete"

Fetch the output from the generated allocation using the nomad alloc logs command.

$ nomad alloc logs 902a3d43
TO: Ada Lovelace <alovelace@programmers.net>
FROM: Charles Babbage <cbabbage@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon.

Sincerely,
Charles
Chief Innovation Officer

Consume a dispatched payload

When defining a Nomad parameterized job, you have the option to use a payload. A payload is a means to supply up to 16Kib of opaque data to instances of your parameterized job during dispatch.

Note

These payloads are included in the dispatched version's job definition data and will consume a corresponding amount of server memory until the dispatched instance of the job is garbage collected.

The parameterized stanza's payload value determines if a payload is required, optional, or forbidden when the job is dispatched.

Nomad writes the dispatch payload into any task containing a dispatch_payload stanza. The path attribute specifies where to write the data relative to the task's local directory.

Enable a mandatory payload

Copy the variables.nomad file to a new file named payload.nomad.

$ cp variables.nomad payload.nomad

Open payload.nomad in your text editor. Add a payload = "required attribute inside the parameterized stanza.

    payload = "required"

Write the payload into the task

Also, add a dispatch_payload stanza inside the output task stanza.

      dispatch_payload {
        file = "payload.txt"
      }

Submit the updated job

Save the payload.nomad file and submit the job to Nomad.

$ nomad job run payload.nomad
Job registration successful

Dispatch the job with the payload

There are two ways to provide a payload to the nomad job dispatch command

from a file
from standard input

Get payload from a file

The tutorial repository contains a sample text file for you to use as the payload—engine.txt

Use the nomad job dispatch command to submit the payload directly

$ nomad job dispatch \
  -meta customer_email="alovelace@programmers.net" \
  -meta customer_name="Ada Lovelace" \
  -meta rep_name="Charles Babbage" \
  -meta rep_email="cbabbage@mechanicalcomputing.com" \
  -meta rep_title="Chief Innovation Officer" \
  -meta product_name="Analytical Engine" \
  template payloads/engine.txt

The command outputs the scheduling information.

Dispatched Job ID = template/dispatch-1620751626-eefe8be3
Evaluation ID     = 297238d6

==> Monitoring evaluation "297238d6"
    Evaluation triggered by job "template/dispatch-1620751626-eefe8be3"
    Allocation "a2971fa2" created: node "f7bc1f2d", group "renderer"
==> Monitoring evaluation "297238d6"
    Allocation "a2971fa2" status changed: "pending" -> "complete" (All tasks have completed)
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "297238d6" finished with status "complete"

Note the allocation ID; retrieve the output of the allocation using the nomad alloc logs command.

$ nomad alloc logs a2971fa2
TO: Ada Lovelace <alovelace@programmers.net>
FROM: Charles Babbage <cbabbage@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon.

Sincerely,
Charles
Chief Innovation Officer

----

CHAPTER VIII
OF THE ANALYTICAL ENGINE

The circular arrangement of the axes of the Difference Engine round large
central wheels led to the most extended prospects. The whole of arithmetic now
appeared within the grasp of mechanism. A vague glimpse even of an Analytical
...

Get payload from standard input

Using standard input allows you submit dynamic payloads to your parameterized jobs. Use - as the filename in the nomad job dispatch command to collect the payload from standard input. This example uses the date command to generate a dynamic payload.

$ date | nomad job dispatch \
  -meta customer_email="alovelace@programmers.net" \
  -meta customer_name="Ada Lovelace" \
  -meta rep_name="Charles Babbage" \
  -meta rep_email="cbabbage@mechanicalcomputing.com" \
  -meta rep_title="Chief Innovation Officer" \
  -meta product_name="Analytical Engine" \
  template -

Dispatched Job ID = template/dispatch-1620751993-b9397497
Evaluation ID     = dc32bd93
 
==> Monitoring evaluation "dc32bd93"
    Evaluation triggered by job "template/dispatch-1620751993-b9397497"
==> Monitoring evaluation "dc32bd93"
    Allocation "b22837a8" created: node "f7bc1f2d", group "renderer"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "dc32bd93" finished with status "complete"

Fetch the output of the dispatched allocation using the nomad alloc logs command.

$nomad alloc logs b22837a8
TO: Ada Lovelace <alovelace@programmers.net>
FROM: Charles Babbage <cbabbage@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon.

Sincerely,
Charles
Chief Innovation Officer

Wed May 12 11:59:30 EDT 2021

Clean up

Clean up the target environment

The cleanup instructions vary depending on your target infrastructure. Select the one that applies.

Nomad dev agent

Stop the Nomad dev agent with Ctrl-C. Nomad dev agents use ephemeral storage and require no further cleanup.

Nomad cluster

Since Nomad clusters maintain persistent state, you will need to perform extra steps to remove all the tutorial artifacts from the cluster.

Remove the parameterized job

Stop the template job. You can optionally pass the -purge flag to tell Nomad to immediately remove it from the server state. Since no scheduling activity is required to remove the job, there is no output generated by the command.

$ nomad job stop -purge template

Remove dispatched instances (optional)

Each time you dispatch a parameterized job, Nomad creates a distinct but related job. These jobs will naturally be removed in time by Nomad's garbage collection process. You can see a list of these instances by using the nomad job status command referencing an ambiguous prefix, like template\. These IDs will be specific to your environment.

Pro Tip - The Nomad command-line tool displays a list of matching IDs whenever an incomplete ID matches more than one item.

$ nomad job status template/
Prefix matched multiple jobs

ID                                     Type   Priority  Status  Submit Date
template/dispatch-1620850743-2d3e0743  batch  50        dead    2021-05-12T16:19:03-04:00
template/dispatch-1620850966-1ba39df6  batch  50        dead    2021-05-12T16:22:46-04:00
template/dispatch-1620851039-1584df83  batch  50        dead    2021-05-12T16:23:59-04:00
template/dispatch-1620851914-68ee4baf  batch  50        dead    2021-05-12T16:38:34-04:00
template/dispatch-1620851954-e0c9f384  batch  50        dead    2021-05-12T16:39:14-04:00

These jobs will naturally be removed in time by Nomad's garbage collection process.

You can also use the nomad job stop command with the -purge flag set to remove individual dispatched instances immediately. For example:

$ nomad job stop -purge template/dispatch-1620748308-41f86ed3
==> Monitoring evaluation "d771b1fc"
    Evaluation triggered by job "template/dispatch-1620748308-41f86ed3"
==> Monitoring evaluation "d771b1fc"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "d771b1fc" finished with status "complete"

Remove the tutorial's working directory

Ensure that you have closed any file editors that might have files in the working directory open.

Change to your home directory and recursively remove the learning environment you created in the first step.

Remove the unarchived learning environment.

$ rm -rf learn-nomad-jobspec-release

Delete the downloaded archive.

$ rm release.zip

Review what you learned

Question 1: How do you create a parameterized job in Nomad?

You create a parameterized job by adding the parameterized stanza to a batch-type Nomad job specification.

Question 2: How do you run an instance of a parameterized job?

You dispatch an instance of a parameterized job by running thennomad job dispatch command or using the Job Dispatch API.

Question 3: How does Nomad supply dispatch variables to the workload?

Nomad dispatch variables are supplied to the workload as environment variables. The environment variable's names are generated by prepending NOMAD_META_ to the variable's name, like NOMAD_META_customer_email.

Question 4: How large can a dispatch payload be? What is an important consideration when using payloads?

Dispatch payloads can be up to 16KiB. Payloads are written directly into the dispatched job's definition, so they use that same amount of server RAM until the job is complete and garbage collected.

Next steps

Nomad parameterized jobs are a powerful tool, encouraging reuse of a job specification by allowing users to provide runtime specific values when dispatching the job. Parameterized jobs enable job creators to make consistent and reusable experiences by encapsulating the specifics of the job and only requiring users to provide the run-time specific information. This ultimately simplifies the user experience for both end-users and API consumers.

In this tutorial, you performed these job specification development actions:

Converted a batch job to a parameterized job.
Added required and optional variables to the job specification.
Defined default values for optional variables
Configured the job to use a dispatch payload.

You performed these end-user operations:

Dispatched instances of a parameterized job
Provided variables during dispatch
Submitted a static and dynamic payload during dispatch

You performed these cluster-operator tasks:

Retrieved logs for an allocation using the nomad alloc logs command

Learn more about the template language used in the .tmpl files in the Learn Go Template Syntax and in the consul-template documentation. You can also learn more about parameterized jobs in the Nomad documentation.

Collection Overview

Job Specifications

Migrate a Linux Java app