Using HashiCorp Vault with Octopus

Mark Harrison

November 21, 2022 • 18 mins

Storing sensitive values in Octopus Deploy solves many problems. If your organization has standardized on a secrets manager, though, that might mean storing sensitive values twice, making secrets management more complicated.

Octopus has supported the concept of sensitive variables since Octopus 2.0, but customers often ask about support for secret managers. One in particular is HashiCorp Vault.

In this post, I walk through a number of HashiCorp Vault step templates we introduced that are designed to retrieve secrets from Vault, for use in your deployments or runbooks.

As of November 2022, our external secret storage templates for HashiCorp Vault are certified by HashiCorp, making Octopus Deploy a certified HashiCorp partner.

In this post

In this post
Introduction
Authentication
Retrieving secrets
- Retrieve KV (v1) secrets step
  - Retrieve KV (v1) secrets parameters
  - Using Retrieve KV (v1) secrets step
- Retrieve KV (v2) secrets step
  - Retrieve KV (v2) secrets parameters
  - Using Retrieve KV (v2) secrets step
Conclusion
Learn more

Introduction

This post assumes you know about custom step templates and the Octopus Community Library. To learn more about these, you can read Ryan Rousseau's two-part series on creating your own step template and publishing it to the library.

In addition, this post doesn't go into detail about Vault server concepts or how to configure a Vault server.

The step templates covered in this post perform both Vault authentication and secret retrieval for both versions 1 and 2 of the Key-Value (kv) Secrets Engine.

All of the step templates make use of the Vault HTTP API so there are no additional dependencies to use them, except connecting to your Vault server. They've all been tested using Vault version 1.11.3, including support for namespaces (a Vault Enterprise feature), and can run on both Windows and Linux with Powershell Core installed.

Authentication

Before interacting with Vault, you must authenticate against an auth method. Vault offers a number of different authentication options. The following step templates have been created to support Vault authentication:

LDAP Login
JWT Login
AppRole Login
AppRole Get wrapped SecretID
AppRole Unwrap SecretID
AppRole Unwrap SecretID and Login

The AppRole method is the recommended way to authenticate with Vault for servers.

Upon authentication with Vault, a token is generated that can be used in further interactions with Vault.

The HashiCorp Vault - LDAP Login step template authenticates with a Vault Server using the LDAP authentication method. This allows Vault integration without duplicating username or password configuration.

You might choose to authenticate using LDAP if you already have an LDAP server available and use service accounts to control access to sensitive information.

Microsoft's Active Directory supports LDAP using Active Directory Lightweight Directory Services.

After authentication, the client_token from the Vault response will be made available as a sensitive output variable named LDAPAuthToken for use in other steps.

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
LDAP Auth Login path: The path that the LDAP method is mounted at. The default is /auth/ldap.
Username: The LDAP username.
Password: The LDAP password.

The LDAP Login step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - LDAP Login].Output.LDAPAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - LDAP Login with the name of your step for any output variable names.

The HashiCorp Vault - JWT Login step template authenticates with a Vault Server using the JWT authentication method. This allows Vault integration using a JSON Web Token.

You might choose to authenticate using a JWT if you're already using this auth method in your organization. It's a good option when making requests over HTTP, as it's a self-contained token that can be generated anywhere.

Generating a JWT
Octopus has 2 existing step templates to generate a JWT, signed with a private key, but they aren't covered in this post. For more information see the step template descriptions:

After authentication, the client_token from the Vault response is made available as a sensitive output variable named JWTAuthToken for use in other steps.

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
JWT Auth Login path: The path that the JWT method is mounted at. The default is /auth/jwt.
JWT role: The Vault JWT role.
JWT Token: The signed JSON Web Token (JWT) to login with.

The JWT Login step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - JWT Login].Output.JWTAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - JWT Login with the name of your step for any output variable names.

The HashiCorp Vault - AppRole Login step template authenticates with a Vault Server using the AppRole authentication method. This is perfect for use with Octopus. HashiCorp recommends it for machines or apps:

This auth method is oriented to automated workflows (machines and services), and is less useful for human operators.

With an AppRole, a machine can log in with:

RoleID, think of this as the username in an authentication pair.
SecretID, think of this as the password in an authentication pair.

Don't store the SecretID:
Storing the RoleID in Octopus as a sensitive variable is a good way to ensure it remains encrypted until required.

However, the same is not recommended for the SecretID.

A SecretID, just like a password is designed to expire. Storing the SecretID could also provide the ability to retrieve all secrets as both the RoleID and SecretID would be available.

We recommend you use the more secure Get wrapped SecretID and Unwrap SecretID and Login step templates, as they use one of the best practices, response wrapping.

If you use the AppRole login step template, we recommend you provide the SecretID at execution time using a sensitive prompted variable.

After it's authenticated, the client_token from the Vault response is made available as a sensitive output variable named AppRoleAuthToken for use in other steps.

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
App Role Path: The path where the approle auth method is mounted.
Role ID: The RoleID of the AppRole.
Secret ID: The SecretID of the AppRole.

The AppRole Login step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Login].Output.AppRoleAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Login with the name of your step for any output variable names.

AppRole best practices

The AppRole authentication method is considered a trusted-broker method. This means the onus of trust rests in the system acting as the authentication intermediary (the broker) between the client (typically an Octopus deployment target) and Vault.

An important best practice is to avoid storing an AppRole SecretID. Instead, use response wrapping to provide a wrapping token that will provide an access mechanism to retrieve a SecretID when required. This method of obtaining a SecretID is also known as a Pull mode as it requires the SecretID to be fetched or pulled from the AppRole.

The Vault documentation contains recommended patterns when using AppRole authentication.

Here's a summary of the recommendations:

Use a secured system to act as the broker for retrieving a wrapped SecretID.
Use response wrapping to obtain a SecretID.
Limit the number of uses and Time-to-live (TTL) value for a SecretID to prevent overuse.
Avoid anti-patterns such as having the broker retrieve secrets.

Secure the broker:
Since the trust rests on the broker, we recommend using the Octopus Server's Built-in Worker, or a highly-secured External Worker to act as the broker. It would be responsible for retrieving a wrapped SecretID and passing that value to the machine (the client) that authenticates with Vault.

To support these recommended practices, three additional AppRole step templates have been created:

AppRole Get Wrapped SecretID
AppRole Unwrap SecretID
AppRole Unwrap SecretID and Login

AppRole Get Wrapped SecretID step

The HashiCorp Vault - AppRole Get Wrapped Secret ID step template generates a response-wrapped SecretID for the AppRole authentication method.

The step template authenticates with a Vault Server using a token to retrieve a wrapped SecretID. The response contains a wrapping token and other metadata such as the creation path for the token.

This value can be used to validate no malfeasance has occurred. The wrapping token can then be used to retrieve the actual SecretID value.

The token used to authenticate to retrieve a wrapped SecretID should be of limited scope and should only be allowed to retrieve wrapped SecretIDs. Consider creating a long-lived Vault token as this presents only a minor risk.

After the response has been received from the Vault server, 2 sensitive output variables are created for use in other steps:

WrappedToken This is the wrapped token from the response, used to retrieve the actual SecretID value.
WrappedTokenCreationPath This is the creation path for the token. It allows you to validate no malfeasance has occurred.

AppRole Get Wrapped SecretID parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
App Role Path: The path where the AppRole auth method is mounted.
Role Name: The role name of the AppRole.
Time-to-live (TTL): The TTL in seconds of the response-wrapping token itself. The default is: 120s
Auth Token: The token used to authenticate with Vault to generate a response-wrapped SecretID.

Using the AppRole Get Wrapped SecretID step

The Get Wrapped SecretID step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variables can be used to validate and retrieve the actual SecretID value:

#{Octopus.Action[HashiCorp Vault - AppRole Get Wrapped Secret ID].Output.WrappedToken}
#{Octopus.Action[HashiCorp Vault - AppRole Get Wrapped Secret ID].Output.WrappedTokenCreationPath}

Tip: Remember to replace HashiCorp Vault - AppRole Get Wrapped Secret ID with the name of your step for any output variable names.

AppRole Unwrap SecretID step

The HashiCorp Vault - AppRole Unwrap Secret ID step template retrieves and unwraps a SecretID for an AppRole using a wrapping token.

The secret_id from the Vault response will be made available as a sensitive output variable named UnwrappedSecretID for use in other steps.

AppRole Unwrap SecretID parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
Wrapped Token: The wrapping token used to retrieve the actual Secret ID from Vault.
Token Creation Path: Optional The creation path for the wrapped token. If this value is provided, the step template will perform a wrapping lookup to validate no malfeasance has occurred.

Using the Unwrap SecretID step

The Unwrap SecretID step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

Note the use of sensitive output variables in the step parameters. In this example, the values are created using the Get Wrapped SecretID step template named HashiCorp Vault - AppRole Get Wrapped Secret ID.

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Unwrap Secret ID].Output.UnwrappedSecretID} can be used to authenticate with Vault and receive a token that can then be used to retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Unwrap Secret ID with the name of your step for any output variable names.

The HashiCorp Vault - AppRole Unwrap Secret ID and Login step template is provided as a convenient way to combine two step templates used with Vault into one:

AppRole Unwrap SecretID: It retrieves and unwraps a SecretID for an AppRole using a wrapping token.
AppRole login: It authenticates with Vault using a supplied RoleID and the unwrapped SecretID.

It's designed as the second part of a two-step workflow with Vault:

Get a wrapped SecretID using the AppRole Get wrapped SecretID step template.
Provide the wrapped SecretID stored in a sensitive output variable from the first step to this step template to unwrap and authenticate.

After it's authenticated, the client_token from the Vault response is made available as a sensitive output variable named AppRoleAuthToken for use in other steps.

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
App Role Path: The path where the AppRole auth method is mounted.
Role ID: The RoleID of the AppRole.
Wrapped Token: The wrapping token used to retrieve the actual Secret ID from Vault.
Token Creation Path: Optional The creation path for the wrapped token. If this value is provided, the step template will perform a wrapping lookup to validate no malfeasance has occurred.

The Unwrap SecretID and Login step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

Note the use of sensitive output variables in the step parameters. In this example, the values were created using the Get Wrapped SecretID step template named HashiCorp Vault - AppRole Get Wrapped Secret ID.

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Unwrap Secret ID and Login].Output.AppRoleAuthToken} can be used to authenticate, and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Unwrap Secret ID and Login with the name of your step for any output variable names.

Retrieving secrets

After you authenticate with Vault, you receive an authentication token that can be used to retrieve secrets. Secrets in Vault are stored in a secrets engine, of which there are many different types.

The step templates created to support retrieving secrets focus on the Key-Value (kv) Secrets Engine as it's a generic Key-Value store used to store arbitrary secrets:

Retrieve KV (v1) secrets step
Retrieve KV (v2) secrets step

Retrieve KV (v1) secrets step

The HashiCorp Vault - Key Value (v1) retrieve secrets step template retrieves one or more secrets stored in a v1 Key-Value secrets engine.

Retrieving a single secret requires:

The path to the secret.
An authentication token with permission to access the secret.
Optionally, a list of field names to retrieve.

An advanced feature of the step template offers support for retrieving multiple secrets at once. This requires changing the Secrets retrieval method parameter to Multiple vault keys.

It's also possible to recursively retrieve secrets. This is useful when you want to retrieve all secrets for a given path.

For each secret retrieved, a sensitive output variable is created for use in subsequent steps. By default, only a count of the number of variables created will be shown in the task log. To see the names of the variables in the task log, change the Print output variable names parameter to True.

Retrieve KV (v1) secrets parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
Auth Token: The token used to authenticate to retrieve secrets.
Secrets Path: The full path to the secret(s) you want to retrieve. The value should include both the path where the secrets engine is mounted, as well as the path to the secret itself.
Secrets retrieval method: Choose between retrieving a single secret or multiple secrets. Retrieving a single secret is the equivalent of a vault kv get command using the Get method. Retrieving multiple secrets is the equivalent of the combination of both a vault kv list command using the List method and then subsequent vault kv get commands for each secret.
Recursive retrieval: If multiple secrets are being retrieved, should any sub-folders also be enumerated and retrieved? The default is: False.
Field names: Choose specific fields to be retrieved from identified secrets. This is useful when you only want to retrieve specific fields from one or more secrets. You can optionally include a name for the output variable.
Print output variable names: Write out the Octopus output variable names to the task log. The default is: False.

Using Retrieve KV (v1) secrets step

The Key Value (v1) retrieve secrets step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

Note the use of the sensitive output variable in the Auth Token parameter. In this example, the value was created using the Unwrap SecretID and Login step template named HashiCorp Vault - AppRole Unwrap Secret ID and Login.

After you fill in the parameters, you can execute the step in a runbook or deployment process. On successful execution, any matching secrets are stored as sensitive output variables. If you configure your step to print the variable names, they appear in the task log:

In subsequent steps, output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace HashiCorp Vault - Key Value (v1) retrieve secrets with the name of your step for any output variable names.

Retrieve KV (v2) secrets step

The HashiCorp Vault - Key Value (v2) retrieve secrets step template retrieves one or more secrets stored in a v2 Key-Value secrets engine.

One of the key advantages of the v2 Key-Value secrets engine is its support for versioned secrets. This is useful if you need to roll back secrets in the event of unintentional data loss.

Retrieving a single secret requires:

The path to the secret
An authentication token with permission to access the secret
Optionally, a list of field names to retrieve

This step template offers advanced features:

Support for retrieving multiple secrets at once. This requires changing the Secrets retrieval method parameter to Multiple vault keys. It's also possible to recursively retrieve secrets. This is useful to retrieve all secrets for a given path.
Support for retrieving a specific version of a secret. This is only supported when retrieving a single secret.

For each secret retrieved, a sensitive output variable is created for use in subsequent steps. By default, only a count of the number of variables created will be shown in the task log. To see the names of the variables in the Task log, change the Print output variable names parameter to True.

Retrieve KV (v2) secrets parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Namespace: Optional The namespace to use. Nested namespaces can be supplied, e.g., ns1/ns2. Note: Namespaces are only supported on Vault Enterprise.
Auth Token: The token used to authenticate to retrieve secrets.
Secrets Path: The full path to the secret(s) you want to retrieve. The value should include both the path where the secrets engine is mounted, as well as the path to the secret itself.
Secrets retrieval method: Choose between retrieving a single secret or multiple secrets. Retrieving a single secret is the equivalent of a vault kv get command using the Get method. Retrieving multiple secrets is the equivalent of the combination of both a vault kv list command using the List method and then subsequent vault kv get commands for each secret.
Recursive retrieval: If multiple secrets are being retrieved, should any sub-folders also be enumerated and retrieved? The default is: False.
Secret Version: Optional When retrieving a single secret, choose the version of the secret to retrieve. For example, if you want version 2 of all field values in a secret, enter the value 2.
Field names: Choose specific fields to be retrieved from identified secrets. This is useful when you only want to retrieve specific fields from one or more secret(s). You can optionally include a name for the output variable.
Print output variable names: Write out the Octopus output variable names to the task log. The default is: False.

Using Retrieve KV (v2) secrets step

The Key Value (v2) retrieve secrets step is added to deployment and runbook processes in the same way as other steps.

After you add the step to your process, fill out the parameters in the step:

In subsequent steps, the output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace HashiCorp Vault - Key Value (v2) retrieve secrets with the name of your step for any output variable names.

Conclusion

The templates covered in this post show how it's possible to extend the functionality of Octopus and retrieve secrets from Vault, or any other secrets manager, and use them in your deployments or runbooks. Be sure to check them out and more in the Secrets Management space in our sample instance.

Learn more

For further information, you can read:

The AppRole Pull Authentication tutorial showing how to retrieve SecretIDs securely
HashiCorp Vault documentation for the K/V v1 Secrets Engine
HashiCorp Vault documentation for the K/V v2 Secrets Engine

Happy deployments!

Using HashiCorp Vault with Octopus

In this post

Introduction