Define a backend for the secrets engine

In these tutorials, you will write a custom secrets engine against the authentication API of a fictional coffee-shop application using the HashiCorp Vault Plugin SDK.

You will learn how to create a new secrets backend, build a set of Vault roles, and create workflows to renew and revoke an API token using Vault.

You may build your own secrets engine to:

• Manage secrets for an existing internal or proprietary tool
• Extend the capabilities of an existing secrets engine (such as scheduled revocation of tokens) with a custom workflow

In this tutorial, you will set up your Vault secrets engine development environment and create a backend to store data, configuration, and secrets.

To do this, you will:

1. Set up your development environment.
   You will clone the HashiCups secrets engine repository. This contains many of the interfaces and objects you need to create a secrets engine.
2. Explore the plugin's entry point.
   You will examine a file that allows Vault to start the secrets engine.
3. Explore the secrets engine's backend.
   You will examine a scaffold that defines an empty schema and functions to store the secrets engine's information in a Vault backend.
4. Explore the client to access the demo application.
   You will examine the client that encapsulates an API client for the secrets engine to use.
5. Explore the configuration for the secrets engine
   You will examine the configuration attributes for the secrets engine, defined at the config path.
6. Update the client to access your target API
   You will replace the code in the scaffold with a client to access the target API.
7. Update the backend to use your target API's client
   You will replace the code in the scaffold to create a secrets engine backend that can access the target API.

Prerequisites

• Golang 1.16+ installed and configured.
• Vault 1.8+ CLI installed locally.

Set up your development environment

Clone the learn-vault-plugin-secrets-hashicups repository.

$ git clone https://github.com/hashicorp-education/learn-vault-plugin-secrets-hashicups

Change into the repository directory.

$ cd vault-plugin-secrets-hashicups

Explore the plugin's entry point

Open cmd/vault-plugin-secrets-hashicups/main.go. The contents of the main function use the Vault SDK's plugin library to start the plugin and communicate with the Vault API.

As you examine the file, note the following:

• VaultPluginTLSProvider: runs inside a plugin and gets the SSL certificate for Vault, returning a Vault TLS configuration object.

• BackendFactoryFunc: provide your secrets engine's backend factory to BackendFactoryFunc. In this case, we pass the factory for the HashiCups backend framework.

• hclog: use HashiCorp's logging package to surface errors from the plugin.

package main

import (
    "os"

    "github.com/hashicorp/go-hclog"
    hashicups "github.com/hashicorp/vault-guides/plugins/vault-plugin-secrets-hashicups"
    "github.com/hashicorp/vault/api"
    "github.com/hashicorp/vault/sdk/plugin"
)

func main() {
    apiClientMeta := &api.PluginAPIClientMeta{}
    flags := apiClientMeta.FlagSet()
    flags.Parse(os.Args[1:])

    tlsConfig := apiClientMeta.GetTLSConfig()
    tlsProviderFunc := api.VaultPluginTLSProvider(tlsConfig)

    err := plugin.Serve(&plugin.ServeOpts{
        BackendFactoryFunc: hashicups.Factory,
        TLSProviderFunc:    tlsProviderFunc,
    })
    if err != nil {
        logger := hclog.New(&hclog.LoggerOptions{})

        logger.Error("plugin shutting down", "error", err)
        os.Exit(1)
    }
}

Explore the secrets engine's backend

Open backend.go. The contents create a backend in Vault for the secrets engine to store data.

The first function, Factory, creates a new backend in Vault as logical.Backend. It should include a call to set up the backend.

func Factory(ctx context.Context, conf *logical.BackendConfig) (logical.Backend, error) {
    b := backend()
    if err := b.Setup(ctx, conf); err != nil {
        return nil, err
    }
    return b, nil
}

The file contains a backend object for your secrets engine. In this case, you name it hashiCupsBackend. It should have three attributes:

1. framework.Backend: implements logical.Backend
2. lock: locking mechanisms for writing or changing secrets engine data
3. client: stores the client for the target API, HashiCups

type hashiCupsBackend struct {
    *framework.Backend
    lock   sync.RWMutex
    client *hashiCupsClient
}

The file creates a method named backend that returns a hashiCupsBackend object. It contains a few important attributes, including:

• Help: outputs help text for the backend.
• PathsSpecial: sets any special paths for the secrets engine's API. Secrets engines usually reference two main fields, with optional fields typically used for auth methods.
  • LocalStorage: For enterprise. These paths store data local to the Vault instance and will not be replicated, more important if you use Vault's Write-Ahead-Log. You would add a path for framework.WALPrefix. You do not need to set it for this secrets engine.
  • SealWrapStorage: For enterprise. When using a seal, these paths should be wrapped with extra encryption. These will usually be set to:
    1. config: configuration of the secrets engine
    2. role/*: Vault role for the secrets engine
  • Root: Optional. Paths that require a root token for access.
  • Unauthenticated: Optional. Paths that can be accessed without authentication. Usually used for for auth methods on the /login endpoint.
• Paths: add the secrets engine's API paths.
• Secrets: add any secrets types, such as tokens or passwords
• BackendType: use logical.TypeLogical for secrets engines
• Invalidate: call the function to invalidate the secrets engine's configuration.

func backend() *hashiCupsBackend {
    var b = hashiCupsBackend{}

    b.Backend = &framework.Backend{
        Help: strings.TrimSpace(backendHelp),
        PathsSpecial: &logical.Paths{
            LocalStorage: []string{},
            SealWrapStorage: []string{
                "config",
                "role/*",
            },
        },
        Paths:       framework.PathAppend(),
        Secrets:     []*framework.Secret{},
        BackendType: logical.TypeLogical,
        Invalidate:  b.invalidate,
    }
    return &b
}

The file includes two methods extending the backend type. Implement a reset method to lock the backend while you reset the target API client object.

func (b *hashiCupsBackend) reset() {
    b.lock.Lock()
    defer b.lock.Unlock()
    b.client = nil
}

You also need to implement the invalidate method referenced in backend. It calls reset to reset the configuration and target API client object.

func (b *hashiCupsBackend) invalidate(ctx context.Context, key string) {
    if key == "config" {
        b.reset()
    }
}

Examine the getClient method in backend.go. The method should take in the context and Vault storage API interface. You may notice that it does not return the demo application client, as per the function. You will update this method when you implement the target API client.

func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    return nil, fmt.Errorf("need to return client")
}

At the bottom of backend.go, define a string with help text for the backend.

const backendHelp = `
The HashiCups secrets backend dynamically generates user tokens.
After mounting this backend, credentials to manage HashiCups user tokens
must be configured with the "config/" endpoints.
`

Explore the client to access the demo application

Open client.go.

Import the Go SDK for HashiCups from a library named hashicups-client-go.

import (
    hashicups "github.com/hashicorp-demoapp/hashicups-client-go"
)

The file initializes an empty hashiCupsClient object that stores a reference to the client to interface with the HashiCups API.

type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    return &hashiCupsClient{nil}, nil
}

newClient takes a Vault configuration object as an input, something you will define depending on the configuration for your secrets engine.

type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    return &hashiCupsClient{nil}, nil
}

Explore the configuration for the secrets engine

Open path_config.go.

Define a constant for configStoragePath as config. You use the config path for secrets engines to define attributes like username, password, and API endpoint.

package secretsengine

import (
    "context"
    "fmt"

    "github.com/hashicorp/vault/sdk/framework"
    "github.com/hashicorp/vault/sdk/logical"
)

const (
    configStoragePath = "config"
)

The file defines an object for hashiCupsConfig. You need this object in order to store configuration attributes related to the secrets engine.

type hashiCupsConfig struct {
    Username string `json:"username"`
    Password string `json:"password"`
    URL      string `json:"url"`
}

Examine the getConfig method. You need to implement this method to pass the context and storage path for Vault to store the hashiCupsConfig into the config path for the secrets engine.

func getConfig(ctx context.Context, s logical.Storage) (*hashiCupsConfig, error) {
    entry, err := s.Get(ctx, configStoragePath)
    if err != nil {
        return nil, err
    }

    if entry == nil {
        return nil, nil
    }

    config := new(hashiCupsConfig)
    if err := entry.DecodeJSON(&config); err != nil {
        return nil, fmt.Errorf("error reading root configuration: %w", err)
    }

    return config, nil
}

Update the client to access your target API

Return to client.go. You must configure the client to access the target API, HashiCups.

Replace the newClient method in the scaffold. The method now checks that hashiCupsConfig has all of its attributes and has a configuration defined.

package secretsengine

import (
    hashicups "github.com/hashicorp-demoapp/hashicups-client-go"
)

type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    if config == nil {
        return nil, errors.New("client configuration was nil")
    }

    if config.Username == "" {
        return nil, errors.New("client username was not defined")
    }

    if config.Password == "" {
        return nil, errors.New("client password was not defined")
    }

    if config.URL == "" {
        return nil, errors.New("client URL was not defined")
    }

    c, err := hashicups.NewClient(&config.URL, &config.Username, &config.Password)
    if err != nil {
        return nil, err
    }
    return &hashiCupsClient{c}, nil
}

Using the hashiCupsConfig object you defined in path_config.go, initialize a new HashiCups client using the URL, username, and password. Return the HashiCups client in the hashiCupsClient object.

package secretsengine

import (
    hashicups "github.com/hashicorp-demoapp/hashicups-client-go"
)

type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    if config == nil {
        return nil, errors.New("client configuration was nil")
    }

    if config.Username == "" {
        return nil, errors.New("client username was not defined")
    }

    if config.Password == "" {
        return nil, errors.New("client password was not defined")
    }

    if config.URL == "" {
        return nil, errors.New("client URL was not defined")
    }

    c, err := hashicups.NewClient(&config.URL, &config.Username, &config.Password)
    if err != nil {
        return nil, err
    }
    return &hashiCupsClient{c}, nil
}

Update the backend to use your target API's client

Return to backend.go. Recall that the getClient method returned an error by default. You need to add your client to this method in order for the secrets engine to access your target API!

Replace the getClient function in backend.go. You must retrieve your secrets engine's configuration using the getConfig function.

func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    config, err := getConfig(ctx, s)
    if err != nil {
        return nil, err
    }

    if config == nil {
        config = new(hashiCupsConfig)
    }

    b.client, err = newClient(config)
    if err != nil {
        return nil, err
    }

    return b.client, nil
}

Vault will keep running the secrets engine and checking for a valid client. If you delete the configuration for the secrets engine or reset it, getClient returns a new hashiCupsConfig.

func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    config, err := getConfig(ctx, s)
    if err != nil {
        return nil, err
    }

    if config == nil {
        config = new(hashiCupsConfig)
    }

    b.client, err = newClient(config)
    if err != nil {
        return nil, err
    }

    return b.client, nil
}

Otherwise, Vault will create a new client based on the configuration in the secrets engine. Set the client to the backend object and return it.

func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    config, err := getConfig(ctx, s)
    if err != nil {
        return nil, err
    }

    if config == nil {
        config = new(hashiCupsConfig)
    }

    b.client, err = newClient(config)
    if err != nil {
        return nil, err
    }

    return b.client, nil
}

Next steps

Congratulations! You configured the secrets engine to retrieve attributes from its config path and create a new client to access your target API.

If you are stuck in this tutorial, refer to the plugins/vault-plugin-secrets-hashicups/solution directory.

• To learn more about Vault plugins, refer to the Vault Plugin System Documentation.
• Define your secrets engine's configuration in the next tutorial.