Users with the exec driver and host volumes

With Nomad's host volume support, you can mount any directory on the Nomad client into an allocation. These directories can be simple filesystem directories on a client, or mounted filesystems like NFS or GlusterFS.

Jobs can ask Nomad to mount these volumes to tasks within a task group.

In this tutorial, you will use a host volume named scratch to provide persistent storage to the job. To provide isolation between the tasks, the job uses the default user for exec jobs—nobody—and a user named user1.

Prerequisites

• Nomad with host volumes support—v0.10.0 or greater
• Linux (any version, any distribution)
• Nomad's exec driver available and enabled

Create the directory structure

Create a directory to back your Nomad host volume on your client node. Set it as your current directory.

$ mkdir -p /opt/nomad/scratch

$ cd /opt/nomad/scratch

Inside that directory, make a directory named nobody.Change the owner of the directory to the nobody user.

$ mkdir nobody

$ chown nobody nobody

Create a user named user1 on the same machine then repeat the previous steps. You may choose to use a different username if you prefer; just remember to stay consistent as you go.

$ useradd -M -U user1

$ mkdir user1

$ chown user1:user1 user1

Remove group and world access to the user1 directory.

$ chmod 700 user1

Configure the host volume

Add the configuration to present this directory as a host volume in your Nomad configuration.

Add the host volume specification inside of your client stanza like this.

client {
  host_volume "scratch" {
    path      = "/opt/nomad/scratch"
    read_only = false
  }
}

If you are using more than one configuration file as suggested in the tip, don't forget to include the surrounding client stanza so that the configuration merges properly.

Restart the nomad service on this node. For example, to restart a service named nomad using systemd, run the following:

$ sudo systemctl restart nomad

Verify the host volume is available

Once Nomad has restarted on the client, use the nomad node status command to verify that the host volume is available. Specify the -verbose flag to view the list of host volumes. If you are still in a shell on the client node, you can use the -self flag rather than providing the node's ID as the parameter.

$ nomad node status -verbose -self

The command prints a lot of useful output. For now, focus on the Host Volumes section. Verify that scratch is present and agrees with the configuration you specified earlier.

...
Host Volumes
Name            ReadOnly  Source
scratch         false     /opt/nomad/scratch
...

Run the job

Fetch the scratch job from Github.

$ wget https://raw.githubusercontent.com/hashicorp/nomad-education-content/main/scratch.nomad

Open the scratch.nomad file in a text editor and take a moment to familiarize yourself with the job. It contains two exec tasks that start simple bash sleep loops and then it mounts the scratch host volume into both of them. This provides just enough environment to connect into in the later steps.

Run the scratch job.

$ nomad run scratch.nomad

Troubleshooting startup

If your output contains a message like the following, verify that your host volume is properly configured and look for typos.

    Task Group "group" (failed to place 1 allocation):
      * Constraint "computed class ineligible": 3 nodes excluded by filter

Nomad jobs using a host volume create an implicit scheduler constraint. This ensures they run on client nodes with the host volume available or wait until one is available. This message indicates that there was no client available that met the constraint.

Interact with the scratch directory

Run nomad status scratch and make note of the Allocation ID from the command output. You need it for running the nomad alloc exec command to interact with the sample environments. Export it into an environment variable named ALLOC_ID for convenience.

This one-liner can collect it for you from Nomad and store it into the variable.

$ ALLOC_ID=$(nomad alloc status -t '{{ range . }}{{if eq .JobID "scratch"}}{{if eq .DesiredStatus "run"}}{{ .ID }}{{end}}{{end}}{{end}}')

Connect to the "nobody" task

Use nomad alloc exec to connect to the "nobody" task.

$ nomad alloc exec -task nobody ${ALLOC_ID} /bin/bash

At the task container's default bash prompt, verify that you are the "nobody" user by running whoami

$ whoami
nobody

Change to the /scratch folder.

$ cd /scratch

Do a long listing of the files in the scratch directory.

$ ls -l
drwxr-xr-x 2 nobody nobody 22 Sep  3 10:38 nobody
drwx------ 2 user1  user1   6 Sep  3 09:28 user1

Verify that you do not have access to the user1 folder using ls and cd.

$ ls user1
ls: cannot open directory user1: Permission denied

$ cd user1
bash: cd: user1: Permission denied

Change into the nobody folder. Use echo and redirection to create a file.

$ cd nobody

$ echo "Hello nobody" > hello.txt

$ cat hello.txt
Hello nobody

Exit the nomad alloc exec session by typing exit.

exit

Connect to the "user1" task

Start a new exec session, this time to the "user1" task.

$ nomad alloc exec -task user1 ${ALLOC_ID} /bin/bash

At the task container's default bash prompt, verify that you are the "user1" user by running whoami

$ whoami
user1

Change to the /scratch folder.

$ cd /scratch

Do a long listing of the files in the scratch directory.

$ ls -l
drwxr-xr-x 2 nobody nobody 22 Sep  3 10:38 nobody
drwx------ 2 user1  user1   6 Sep  3 09:28 user1

Since the nobody folder allows for read access, read the file you created earlier.

$ cat nobody/hello.txt
Hello nobody

Change into the nobody folder and verify that user1 does not have write access.

$ cd nobody

$ touch test.txt
touch: cannot touch ‘test.txt’: Permission denied

Change back to the /scratch/user1 folder. Create a file here using echo and redirection.

$ cd /scratch/user1

$ echo "I am user1" > user1.txt

$ cat user1.txt
I am user1

Exit the nomad alloc exec session by typing exit.

exit

Verify persistence

Stop the job and restart it.

$ nomad stop scratch

$ nomad run scratch.nomad

Fetch its new allocation ID into your ALLOC_ID environment variable

$ ALLOC_ID=$(nomad alloc status -t '{{ range . }}{{if eq .JobID "scratch"}}{{if eq .DesiredStatus "run"}}{{ .ID }}{{end}}{{end}}{{end}}')

Connect to the "user1" task

Start a new exec session to the "user1" task.

$ nomad alloc exec -task user1 ${ALLOC_ID} /bin/bash

At the task container's default bash prompt, verify that you are the "user1" user by running whoami

$ whoami
user1

Use the cat command to output the files that you created earlier in the scratch folder.

$ cat /scratch/nobody/hello.txt
Hello nobody

$ cat /scratch/user1/user1.txt
I am user1

Exit the nomad alloc exec session by typing exit.

exit

Clean up

To clean up after this tutorial, do the following.

Stop the "scratch" job

$ nomad job stop scratch

Remove tutorial artifacts from Nomad client

If you are no longer in a shell session to your Nomad client, connect to the Nomad client you created and configured the host volume on.

Remove the host volume configuration; restart the Nomad client process

Edit the Nomad configuration to remove the host volume stanza or additional configuration file you created earlier. Restart the Nomad process.

Verify that the host volume doesn't appear in the node status output.

$ nomad node status -self -verbose

Remove the folder backing the host volume

Delete the folder that you created to back the host volume.

$ rm -rf /opt/nomad/scratch

Remove the tutorial user

$ userdel user1

Dig deeper

In this tutorial, you used Nomad's ability to set an exec task's user context to provide finer grained access permissions to a directory backing a Nomad host volume. You can use this technique to reduce the number of host volume configurations that you need to manage in your Nomad configuration.

Learn more about the key concepts used in this tutorial.

• host_volume configuration stanza
• The Nomad Job Specification - user
• The nomad alloc exec command