Vault Agent Windows service

All authenticated requests to Vault must include a valid token, which means that Vault clients must first authenticate with Vault, and receive a token.

The Secure Introduction of Vault Clients tutorial details available approaches for solving the secret zero problem.

One comprehensive and robust way to solve this problem is to use Vault Agent. Vault Agent can also manage the lifecycle of additional secrets beyond secret zero.

Challenge

Vault Agent is most effective when available to the client application or user environment on a continuous basis and operated as a system service instead of a foreground user process.

Solution

You can configure Vault Agent to operate as a Windows service to automatically start when Windows starts, and become available for use when the client application or user needs it.

Scenario introduction

You can register Vault as a Windows service in different ways. You will use either the Service Control tool, sc.exe or the New-Service cmdlet to register Vault Agent as a Windows service in this tutorial.

For this scenario, you will use PowerShell sessions and command line invocations to start a Vault dev mode server and configure the AppRole auth method for use by Vault Agent.

You will then configure Vault Agent and register it as a service.

You will also confirm that the Agent service can start and authenticate with the dev mode Vault server.

Prerequisites

You need the following to complete this section of the tutorial:

• A Windows host (Tutorial last tested with Windows Server 2019).
• An Administrator level user account on the Windows host that has the capability to register system services. Follow this tutorial while logged in as the local administrator account.
• Vault version 1.7 or later; you can follow the Install Vault tutorial to install Vault on Windows. The community edition is suitable for completing the example scenario.
• jq for consuming Vault JSON formatted output and capturing specific fields for output to files; you need jq to follow specific steps in the example scenario.

Lab setup

Configure Vault

1. Vault Agent needs an enabled auth method for automatically authenticating to Vault. Enable the AppRole auth method.

   $ vault auth enable approle
   
   Success! Enabled approle auth method at: approle/
   

   The success message indicates that you enabled the AppRole auth method at the default path approle/.

2. Create a role named vault-agent-role for later use by the Agent.

   $ vault write auth/approle/role/vault-agent-role secret_id_ttl=90m token_num_uses=10 token_ttl=60m token_max_ttl=120m secret_id_num_uses=20
   
   Success! Data written to: auth/approle/role/vault-agent-role
   

   Note

   The restrictions on token use from this example role emphasize the importance of short lived secrets.

   In the vault-agent-role, the AppRole secret ID has a 90 minute time to live (TTL) and you can use it 20 times. You can use tokens acquired from this role 10 times, with a maximum TTL of 2 hours.

   Vault Agent will use this role by passing a role ID and secret ID when it performs the automatic authentication.

Configure Vault Agent

In this scenario, Vault Agent will execute with an example configuration instructing it to connect to the Vault dev mode server and authenticate using the AppRole auth method.

You need to write the vault-agent-role role ID and a matching secret ID to files in the Vault agent directory to authenticate with Vault using the AppRole auth method. When the Agent service starts, it will consume these files as part of its automatic authentication functionality.

1. From your existing PowerShell session, create a top-level directory to contain the configuration and other data created when following the example scenario.

   $ New-Item -path c:\vault-agent -ItemType "directory"
   
   Directory: C:\
   
   Mode                LastWriteTime         Length Name
   ----                -------------         ------ ----
   d-----        3/18/2021   8:35 AM                vault-agent
   

   Note

   These examples use the path C:\vault-agent, but you can use any name and have access to, so long as you substitute it for any reference of the scenario directory name.

2. Write the role ID to file in the scenario directory called agent-role-id.

   $  vault read auth/approle/role/vault-agent-role/role-id -format=json | jq -r '.data.role_id' | Out-File -encoding ascii C:\vault-agent\agent-role-id -NoNewline
   

3. Write the secret ID to file in the scenario directory called agent-secret-id.

   $ vault write -f auth/approle/role/vault-agent-role/secret-id -format=json | jq -r '.data.secret_id' | Out-File -encoding ascii C:\vault-agent\agent-secret-id -NoNewline
   

4. Check the file contents, keeping in mind that this will reveal the secret ID value.

   $  Get-Content c:\vault-agent\agent-role-id ; Get-Content c:\vault-agent\agent-secret-id
   
   28a05ccc-99bd-babd-9be0-29e890c279f3
   8f941d1a-ed3b-bd3b-a939-2836b633e541
   

5. The Vault Agent service uses a configuration file to connect to the Vault server. Assign the file content to a variable named AgentConfiguration.

   $AgentConfiguration = @"
   pid_file = "/vault-agent/agent.pid"
   
   vault {
     address = "${env:VAULT_ADDR}"
   }
   
   auto_auth {
     method "approle" {
       config = {
         role_id_file_path   = "/vault-agent/agent-role-id"
         secret_id_file_path = "/vault-agent/agent-secret-id"
       }
     }
   
     sink "file" {
       config = {
         path = "/vault-agent/agent-token"
       }
     }
   }
   
   cache {
     use_auto_auth_token = true
   }
   
   listener "tcp" {
     address     = "127.0.0.1:8100"
     tls_disable = true
   }
   "@
   

   If you have Vault binary version 1.13.0 or later, you can specify a file to write Vault Agent log messages.

   $AgentConfiguration = @"
   pid_file = "/vault-agent/agent.pid"
   
   log_file = "/vault-agent/agent-trace.log"
   
   vault {
     address = "${env:VAULT_ADDR}"
   }
   
   auto_auth {
     method "approle" {
       config = {
         role_id_file_path   = "/vault-agent/agent-role-id"
         secret_id_file_path = "/vault-agent/agent-secret-id"
       }
     }
   
     sink "file" {
       config = {
         path = "/vault-agent/agent-token"
       }
     }
   }
   
   cache {
     use_auto_auth_token = true
   }
   
   listener "tcp" {
     address     = "127.0.0.1:8100"
     tls_disable = true
   }
   "@
   

   A log file named vault-agent-{timestamp}.log will be generated. To see the full list of available parameters, refer to the Vault Agent documentation.

6. Write the AgentConfiguration variable contents to the Vault Agent configuration file.

   $ Set-Content C:\vault-agent\vault-agent.hcl $AgentConfiguration
   

7. Confirm the Vault address in the Agent configuration file.

   VaultHCP Vault Dedicated

   $ Get-Content C:\vault-agent\vault-agent.hcl | Select-String -Pattern 'address'
   

   The first line is for your Vault server - http://127.0.0.1.8200. The second line is the listener address for the Vault Agent - 127.0.0.1:8100.

   address = "http://127.0.0.1:8200"
   address = "127.0.0.1:8100"
   

   $ Get-Content C:\vault-agent\vault-agent.hcl | Select-String -Pattern 'address'
   

   The first line is for your Vault server matching the public address retrieved from the HCP Portal ending with port 8200. The second line is the listener address for the Vault Agent - http://127.0.0.1:8100.

   address = "https://vault-cluster.vault.1b3d5f-1234-5678-ba99-s3asam3str.aws.hashicorp.cloud:8200"
   address = "127.0.0.1:8100"
   

8. Unset existing environment variables.

   VaultHCP Vault Dedicated

   $ Remove-Item Env:\VAULT_ADDR ; Remove-Item Env:\VAULT_TOKEN
   

   $ Remove-Item Env:\VAULT_ADDR ; Remove-Item Env:\VAULT_TOKEN ; Remove-Item Env:\VAULT_NAMESPACE
   

9. Verify removal of the environment variables.

   $ Get-Item Env:
   

   You are now prepared to register the Agent service.

Register Vault Agent as a Windows service

One way to register the service is to use Service Control.

Service Control works best if the path to your Vault binary and the associated Agent configuration file do not contain space characters. Service Control can be difficult to configure if your path has spaces, as you must quote paths containing spaces, and escaping quotes is often non-trivial.

Another alternative is to use the New-Service cmdlet. New-Service is more tolerant about escaping quotes in paths containing spaces. It can sometimes be easier to configure if your path has spaces or you prefer not to use Service Control.

Choose the method you prefer to learn about, and register the Vault Agent service.

$ sc.exe create VaultAgent binPath="C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -config=C:\vault-agent\vault-agent.hcl" displayName="VaultAgent" start=auto
[SC] CreateService SUCCESS

$ sc.exe create VaultAgent binPath="C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -namespace=admin -config=C:\vault-agent\vault-agent.hcl" displayName="VaultAgent" start=auto
[SC] CreateService SUCCESS

$ New-Service -Name "VaultAgent" -BinaryPathName "C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -config=C:\vault-agent\vault-agent.hcl" -DisplayName "VaultAgent" -StartupType "Automatic"

$ New-Service -Name "VaultAgent" -BinaryPathName "C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -namespace=admin -config=C:\vault-agent\vault-agent.hcl" -DisplayName "VaultAgent" -StartupType "Automatic"

$ New-Service -Name "VaultAgent" -BinaryPathName '"C:\my dir\vault.exe" agent -config="C:\my dir\agent-config.hcl"' -DisplayName "Vault Agent" -StartupType "Automatic"

Start Vault Agent service

You can start the service with Service Control, the Start-Service cmdlet, or with the UI in the Windows Service Manager.

$ sc.exe start VaultAgent

SERVICE_NAME: VaultAgent
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 2  START_PENDING
                                (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x7d0
        PID                : 4728
        FLAGS              :

$ Start-Service -Name "VaultAgent"

Verify the Agent token

If the Agent is operating properly, it will automatically authenticate with Vault using AppRole and get a token that it will write to the file sink at C:\vault-agent\agent-token.

Using this token ID value and the Vault Agent address, you can further authenticate with Vault.

Set the Vault Agent address as the value to the VAULT_ADDR environment variable so that you are communicating directly to the Agent and not the Vault dev mode server.

$ $env:VAULT_ADDR="http://127.0.0.1:8100"

Capture the token contained in the agent-token file as the VAULT_TOKEN environment variable value.

$ $env:VAULT_TOKEN = Get-Content C:\vault-agent\agent-token -Raw

Try a token lookup.

$ vault token lookup

Example output:

Key                  Value
---                  -----
creation_time        1616100193
creation_ttl         1h
display_name         approle
entity_id            c0a7378e-1d42-dade-dfb0-d6d448228008
expire_time          2021-03-18T15:25:13.1964324-07:00
explicit_max_ttl     0s
issue_time           2021-03-18T13:43:13.1406039-07:00
last_renewal         2021-03-18T14:25:13.1964324-07:00
last_renewal_time    1616102713
meta                 map[role_name:vault-agent-role]
num_uses             7
orphan               true
path                 auth/approle/login
policies             [default]
renewable            true
ttl                  20m22s
type                 service

Note that the role_name contained in the meta is vault-agent-role, and the token has just the default policy attached. This is confirmation that the token originated from the AppRole auth method on the dev server.

IMPORTANT: VAULT_ADDR usage

In this tutorial, you set the VAULT_ADDR environment variable to the target Vault address. Keep in mind that Vault Agent has its own client for Vault server connections.

This is how Vault Agent transparently proxies connections from the CLI to the Vault server. If you set a VAULT_ADDR environment variable, the Vault Agent will use that value to connect to Vault. This can create an infinite loop where Vault Agent uses the value of VAULT_ADDR to make a connection, and ends up trying to connect to itself instead of the server.

When the connection fails, the Vault Agent increments the port and tries again. The agent repeats these attempts, which leads to port exhaustion.

This problem is a result of the precedence order of the 3 different ways to configure the Vault address. They are, in increasing order of priority:

1. Configuration files
2. Environment variables
3. CLI flags

How to avoid infinite loop problem

1. Do NOT set VAULT_ADDR globally in the environment so that the Vault Agent will use the value in its configuration file to connect to the target Vault server instead. This means that you need to manually set VAULT_ADDR for any sessions involving the Vault CLI.

2. If VAULT_ADDR is set globally in the environment, add the -address="<server address>" CLI flag to the command invocation when creating the Windows service (specified as binPath during service creation). This CLI flag will override both the configuration file and the environment variable, and ensure that Vault Agent is always talking to the correct server.

   Example command:

   $ New-Service -Name "VaultAgent" -BinaryPathName '"C:\my dir\vault.exe" agent -config="C:\my dir\agent-config.hcl" -address="https://production-vault-server:8200"' -DisplayName "Vault Agent" -StartupType "Automatic"
   

Next steps

You have learned how to configure Vault Agent as a Windows service and confirm that the service registers and starts.

You showed that the Vault Agent could connect to the Vault dev mode server, and authenticate by checking for a Vault token at the configured sink location.

You also used the Agent and token ID value to authenticate with Vault as a secondary confirmation that Agent acquired a token from the AppRole auth method.

From here, you can expand your Vault Agent knowledge, and learn about more advanced features, such as Agent Caching and Agent Templates.

Help and reference

• Blog post: Why Use the Vault Agent for Secrets Management?
• Video: Streamline Secrets Management with Vault Agent and Vault 0.11
• Vault Agent
• sc.exe create
• New-Service