Essential HashiCorp Vault Guide and Commands

Introduction to HashiCorp Vault: A Comprehensive Guide and Essential Commands

In modern cloud-native architectures and distributed systems, securing sensitive information such as API keys, database credentials, and tokens is crucial to maintaining data integrity and preventing breaches. HashiCorp Vault offers a comprehensive solution to store, manage, and protect these secrets, providing robust features like dynamic secrets, encryption as a service, and fine-grained access control.

Designed to tackle today’s security challenges, Vault ensures data is encrypted and access to sensitive information is strictly controlled, whether you're dealing with on-premises systems, cloud-based services, or hybrid environments. It integrates seamlessly into modern infrastructures, automating security tasks and reducing human error—two of the most common causes of security incidents.

This article will guide you through essential Vault commands and configurations, covering topics like initializing Vault, managing secrets, auditing access, and resetting Vault in case of failures. By the end of this guide, you'll have a comprehensive understanding of Vault’s command-line interface (CLI) and advanced administrative tasks, along with best practices for deploying Vault in production environments.

Overview of HashiCorp Vault

What is HashiCorp Vault?
Benefits of Using Vault
Secrets Management
Core Features of Vaul

Vault Architecture

Server and Client Architecture
Storage Backend
Secret Engines
Authentication Methods
Audit Devices

HashiCorp Vault Policies and Paths

What is a Path in Vault?
Understanding Policies in Vault
How Policies and Paths Work Together
Best Practices for Using Policies and Paths

Use Cases for HashiCorp Vault

Secrets Management
Dynamic Secrets for Databases
Encryption as a Service
Identity Management and Access Control
Secure Introduction and Authentication
Backend Storage

HashiCorp Vault: Seal and Unseal Concepts

Sealed State
Unseal Keys
Unsealing Process
Why Use Unseal Keys and Sealed State?
Auto-Unseal Feature (Enterprise)

HashiCorp Vault Versions

Vault Open Source
Vault Enterprise
Vault Cloud (HCP Vault)
Is Vault Free to Use in Production?

Basic Vault Commands

Start Vault
Initializing Vault
Unsealing Vault
Logging into Vault

Storing and Retrieving Secrets

The KV secrets engine
KV engine version
KV engine version: a closer look
Enabling and configuring KV v2
Listing secrets
Revoking Access
Dynamic secrets

Vault Token and Token accessor

Vault Token (Access Token)
Token Accessor

Intermediate and Advanced Vault Commands

Enabling Secret Engines
Tuning Vault Policies
Creating and Managing Tokens
Auditing Vault

Resetting HashiCorp Vault Completely

Sealing Vault for Safety
Removing Data from Vault
Deleting Backend Data Storage
Vault Server Configuration Reset
Re-initializing Vault

Best Practices for Vault Administration

Follow the Principle of Least Privilege
Securely Store and Rotate Unseal Keys
Backup Vault Data Regularly
Minimize Root Token Usage
Enable Auditing
Automate Unsealing with Auto-Unseal
Test Policies in a Staging Environment
Leverage Dynamic Secrets

Conclusion
References

Overview of HashiCorp Vault

In today’s digital world, organizations face the complex challenge of securely managing a vast array of sensitive information, including passwords, API keys, encryption keys, database credentials, and other secrets. Traditional methods of storing and managing this data, such as hardcoding secrets in application code or storing them in configuration files, expose organizations to significant security risks. Unauthorized access to secrets can lead to data breaches, compromised systems, and loss of sensitive information, with potentially devastating consequences.

To address these challenges, HashiCorp introduced Vault, a tool specifically designed to provide a secure, centralized solution for managing secrets. Vault is more than just a simple password manager; it’s a comprehensive system that helps organizations control access to secrets, manage their lifecycle, and ensure they are used securely within applications and infrastructure. Let’s delve deeper into the key concepts, features, and benefits of Vault.

What is HashiCorp Vault?

At its core, HashiCorp Vault is a secrets management tool that provides a secure way to store and control access to sensitive information. It acts as a centralized repository where secrets can be stored, accessed, and managed securely. Vault is designed to be highly flexible, supporting a wide range of secret types and offering a variety of ways to integrate with different systems, applications, and environments.

Vault’s primary goal is to minimize the risk associated with managing secrets by providing a unified interface for secret management, enforcing access controls, and automating the process of secret generation and revocation. This makes Vault a critical component of modern security architectures, particularly in environments where security is paramount.

Benefits of Using Vault

Implementing Vault offers several benefits for organizations looking to enhance their security posture:

Centralized Secret Management

By centralizing secret management, Vault eliminates the need to manage secrets in disparate systems, reducing the risk of secrets being mismanaged or exposed.

Improved Security

Vault’s dynamic secrets, encryption, and fine-grained access controls significantly improve the security of sensitive data, reducing the likelihood of unauthorized access.

Compliance and Auditing

Vault’s robust auditing capabilities make it easier for organizations to meet compliance requirements and provide detailed logs for forensic analysis.

Scalability

Vault’s architecture is designed to scale, making it suitable for organizations of all sizes, from small teams to large enterprises with complex infrastructure needs.

Flexibility

Vault’s modular design allows it to be customized to fit a wide range of use cases, making it a versatile tool for different types of organizations and environments.

Enhancing Security with HashiCorp Vault

Summary of Vault security improvements:

1. Hardcoded Secrets

Risk: Hardcoding secrets (API keys, database passwords, etc.) in source code or configuration files is a major security vulnerability. If source code is compromised (e.g., via a public repository), attackers gain access to all embedded secrets.

Vault Solution: Vault avoids hardcoded secrets by offering an API that applications use to fetch secrets at runtime. Secrets can be stored in Vault securely and accessed only when needed.

2. Static and Long-Lived Credentials

Risk: Static credentials that are long-lived (e.g., hardcoded API keys, passwords, or database credentials) are more vulnerable to being compromised since they often remain unchanged for a long time.

Vault Solution: Vault offers Dynamic Secrets that are created on-demand and have a limited Time-To-Live (TTL). These short-lived credentials reduce the exposure risk and limit the damage in case they are compromised, as they automatically expire after a specified TTL.

3. Unauthorized Access to Secrets

Risk: If access to secrets is not properly controlled, unauthorized users or applications might gain access, leading to potential data breaches.

Vault Solution: Vault enforces strict access control using Identity and Policy Management:

Authentication Methods: Vault integrates with various authentication systems (e.g., LDAP, AWS IAM, Kubernetes) to authenticate clients.

Access Control Policies: Vault uses ACL policies to enforce fine-grained access to secrets. Only authorized users or applications can access specific secrets, and each secret can have different policies.

4. Secrets Sprawl

Risk: Secrets sprawl occurs when secrets are scattered across multiple services, files, and environments without centralized management, making them difficult to track and secure.

Vault Solution: Vault provides a centralized storage solution to manage secrets. All secrets are stored in one secure place, with robust access control, audit logging, and encryption, reducing the risk of secrets spreading across multiple locations.

5. Plaintext Secrets in Storage and Transit

Risk: Storing secrets in plaintext (unencrypted) exposes them to attackers if the storage system is compromised. Similarly, if secrets are transmitted without encryption, they could be intercepted.

Vault Solution: Vault ensures secrets are always encrypted at rest and in transit:

Encryption at Rest: Vault encrypts secrets before storing them in its internal storage using strong encryption algorithms (e.g., AES-256).

Encryption in Transit: Vault uses TLS to ensure that all communications between clients and Vault are encrypted, protecting secrets from interception.

6. Man-in-the-Middle (MITM) Attacks

Risk: During transmission, attackers can intercept and manipulate the communication between applications and services, gaining access to secrets or sensitive information.

Vault Solution: Vault uses TLS/SSL to encrypt all communication between the client and the Vault server, preventing unauthorized interception. Mutual TLS authentication can also be used to verify the identity of both the client and server, mitigating MITM attacks.

7. Privileged Escalation and Misuse of Secrets

Risk: If a user or application has unrestricted access to secrets, there is a higher risk of misuse, such as accidental data exposure or malicious activity. Additionally, an attacker who gains access to a privileged account can compromise all secrets.

Vault Solution: Vault implements the principle of least privilege using fine-grained access policies and short-lived tokens that limit what users and applications can do. Each token can be configured to allow access only to specific secrets and expire after a specific TTL, reducing the risk of misuse.

8. Uncontrolled Secret Rotation

Risk: Failing to rotate secrets regularly leaves applications vulnerable because compromised secrets are still valid. Manual rotation can be error-prone and lead to service disruptions.

Vault Solution: Vault can automate the rotation of secrets through Dynamic Secrets and Secret Engine Rotation:

Dynamic Secrets: Vault generates dynamic credentials for databases or cloud services with a limited TTL, automatically revoking or rotating credentials when they expire.

Periodic Rotation: For static secrets (like certificates), Vault can periodically rotate secrets according to a defined schedule, ensuring that secrets are updated without manual intervention.

9. Credential Sharing and Overuse

Risk: When credentials are shared among multiple services or users, it becomes challenging to track usage, making auditing and incident response difficult.

Vault Solution: Vault creates unique, per-use credentials for users or applications, which makes it easier to audit who accessed which secret and when. Audit logging is an important feature that Vault offers, recording every request and response, allowing the detection of anomalous or malicious activity.

10. Exposure of Root Credentials

Risk: Root credentials or master keys can provide unrestricted access, and if compromised, the attacker has full control over the system.

Vault Solution: Vault uses Shamir's Secret Sharing to split the root key into several shares, which need to be combined to unseal the Vault. This process avoids exposing a single root key. Additionally, root tokens should only be used for setup, and Vault enforces the use of limited-privilege tokens for ongoing operations.

11. Service-to-Service Authentication Without Centralized Trust

Risk: When services need to authenticate each other, doing so without a centralized trust mechanism can lead to inconsistencies and vulnerabilities.

Vault Solution: Vault provides trusted identity-based authentication for services through various authentication methods like Kubernetes, AWS IAM, etc., ensuring a consistent and centralized approach for authenticating and authorizing services securely.

12. Certificate Management Issues

Risk: Improper handling of TLS certificates, such as using expired certificates or hardcoding them, can lead to security vulnerabilities.

Vault Solution: Vault offers a PKI (Public Key Infrastructure) Secret Engine that allows for dynamic generation, signing, and revocation of certificates. Vault can also handle automatic renewal of certificates, avoiding the risks associated with expired or hardcoded certificates.

Secrets Management

HashiCorp Vault is a tool designed to protect secrets and other sensitive data by encrypting it and controlling access through policies. At its core, Vault addresses three critical security concerns:

Secrets Management: Store and manage secrets (API keys, passwords, database credentials) securely and with ease.
Data Encryption: Encrypt and decrypt sensitive data on-demand through a unified API, simplifying encryption for applications.
Access Control: Allow fine-grained, role-based access to secrets, defining who can access what data and under what conditions.

Vault supports dynamic secrets, which are generated on-demand and have short lifetimes, reducing the risk of credential leakage. For example, Vault can generate temporary database credentials that expire after a predefined period.

Core Features of Vault

Vault offers a rich set of features that address various aspects of secret management, making it a comprehensive solution for organizations of all sizes. Some of the core features include:

Secret Storage

Vault provides a secure storage backend where secrets are stored in encrypted form. Secrets can be anything from simple text values like passwords to more complex data like JSON objects. The storage backend is pluggable, meaning Vault can integrate with different storage solutions, such as Consul, etcd, MySQL, and more, depending on the organization’s needs.

Dynamic Secrets

One of Vault’s most powerful features is its ability to generate secrets dynamically. Instead of storing static credentials, Vault can create secrets on-the-fly when requested. For example, Vault can generate database credentials that are valid for a short period and automatically revoke them after use. This reduces the risk of credentials being compromised and limits the potential damage if they are exposed.

Encryption as a Service

Vault provides a secure API for encrypting and decrypting data, allowing applications to offload cryptographic operations to Vault. This ensures that sensitive data is encrypted using strong, centrally managed keys, and that encryption practices are consistent across the organization.

Access Control and Policies

Vault uses a fine-grained access control system based on policies. Administrators can define policies that specify who can access certain secrets and what actions they can perform. These policies are enforced by Vault’s authentication methods, which include tokens, LDAP, AWS IAM, and more, ensuring that only authorized users and applications can access sensitive data.

Auditing and Logging

Security and compliance are critical concerns for most organizations, and Vault addresses these with robust auditing and logging capabilities. Vault logs all access and actions performed on secrets, providing a detailed audit trail that can be used for compliance reporting and forensic analysis in case of a security incident.

High Availability and Scalability

Vault is designed to be highly available and scalable, making it suitable for use in large, distributed environments. It supports clustering, where multiple Vault nodes can share the same storage backend and handle requests in an active-active or active-passive configuration. This ensures that Vault can handle large volumes of requests and remains available even in the event of node failures.

HashiCorp Vault works like a hotel check-in, where a client follows a process to verify their identity and access a room.

The functioning of HashiCorp Vault can be compared to a scenario where a client checks into a hotel to book a room, following a specific process to authenticate their identity and gain access. Here’s how this parallel works:

The Hotel (Vault Server): The hotel represents the Vault server. It is the place where important things are securely stored, just like a hotel provides a room for its guests to keep their belongings safe.

The Client Arrives (User Requests Access): The client, or user, approaches the hotel reception (Vault server) to request a room, just as a user might approach Vault to request access to a secret or service.
Document Requested (Authentication): The hotel receptionist asks the client for an identification document, like a passport or ID card, to verify their identity. Similarly, when a user interacts with Vault, they must first authenticate by presenting credentials, such as a token, username/password, or an API key, to prove their identity.
Identity Verification (Authentication Process): The hotel receptionist checks the ID to ensure that the client is who they claim to be. In Vault, this step involves verifying the user’s credentials via the authentication backends (e.g., LDAP, GitHub, or Tokens). If the credentials are valid, Vault knows that the user is authorized.
Room Key (Access Token): Once the receptionist confirms the client's identity, they hand over a key that grants access to the room. Similarly, after successful authentication in Vault, the system issues an access token to the user. This token acts like the hotel room key, allowing the user to perform certain actions (such as retrieving secrets or generating credentials) based on their assigned permissions.
Accessing the Room (Accessing Secrets or Services): The client uses the room key to unlock and enter their hotel room, where they can securely store their belongings. In Vault, the user uses their token to access the resources (e.g., secrets or dynamic credentials) they are authorized for, such as database passwords or API keys.
Key Expiry (Token TTL): Hotel room keys are often only valid for the duration of the guest’s stay and will expire after checkout. Similarly, Vault tokens have a time-to-live (TTL), after which they expire, ensuring limited-time access to the secrets or services. When the token expires, the user must reauthenticate to get a new one, just like the hotel guest would need to check in again if their key expires.
Room Checkout (Revoking Access): If the guest checks out early or their stay ends, the room key becomes invalid, and they can no longer access the room. In Vault, access tokens can be revoked manually or automatically based on the policies in place, terminating the user’s ability to access resources, just as the hotel locks the room when the guest checks out.

This parallel demonstrates the flow of authentication, authorization, and access management in Vault, mirroring the familiar process of checking into a hotel and receiving a room key.

Vault Architecture

Understanding the architecture of Vault is key to grasping how it operates and integrates within an organization’s infrastructure. The architecture of Vault is built around several core components:

Server and Client

Vault operates in a client-server model. The Vault server is the central component that stores and manages secrets, enforces policies, and performs cryptographic operations. Clients interact with the Vault server via its API to store, retrieve, and manage secrets.

Storage Backend

The storage backend is where Vault persists its data, including secrets, encryption keys, and policies. Vault does not encrypt secrets by default in the backend; instead, it encrypts them using a master key that is generated during initialization. The storage backend can be anything from a file system to a distributed system like Consul or etcd.

Secret Engines

Vault’s secret engines are plugins that handle specific types of secrets. Each secret engine is responsible for managing a particular kind of secret, such as key-value pairs, database credentials, or cloud provider credentials. Secret engines are modular, meaning you can enable or disable them based on your specific use case.

Authentication Methods

Authentication methods in Vault are the mechanisms by which users and applications prove their identity before accessing Vault. Vault supports various authentication methods, including token-based authentication, LDAP, AWS IAM, and more. Each method can be configured with specific policies that dictate what a user or application can do within Vault.

Audit Devices

Audit devices are responsible for capturing logs of all requests made to Vault. These logs can be sent to various destinations, such as files, syslog, or third-party logging services. Auditing is crucial for maintaining security and compliance in environments where secrets management is critical.

Sure! Here's an article outline explaining policies and paths in HashiCorp Vault for your blog:

HashiCorp Vault Policies and Paths

What is a Path in Vault?

In HashiCorp Vault, a path is essentially a URL that defines where secrets, authentication methods, or system settings are stored or accessed. Vault uses a hierarchical structure for these paths, which allows granular control over what data can be accessed and by whom. For instance:

`secret/myapp` is a path where secrets related to "myapp" are stored.

`auth/userpass/login` is the path used to authenticate user credentials using the userpass authentication method.

Every interaction with Vault—whether storing, retrieving, or managing secrets—happens through these paths. This path-based architecture allows for highly configurable access controls, as each path can have unique policies governing what operations are permitted.

Understanding Policies in Vault

A policy in HashiCorp Vault defines the permissions granted to users, applications, or services interacting with Vault. Policies control what actions can be performed on specified paths, such as `read`, `write`, `list`, or `delete`.

Policies are written in HCL (HashiCorp Configuration Language) or JSON format and are crucial for maintaining security and access control in Vault. By carefully defining policies, administrators can enforce the principle of least privilege—ensuring users have access only to what they need and nothing more.

Here’s an example of a basic policy in HCL:

path "secret/myapp/*" { capabilities =["read", "list"] }

This policy grants users the ability to read and list all secrets under the `secret/myapp/` path. Policies can be as granular as needed, giving you full control over which users or services can access specific paths and what they are allowed to do.

How Policies and Paths Work Together:

Policies and paths form the backbone of Vault’s access control system. When a user or application interacts with a path (e.g., trying to retrieve a secret), Vault checks the associated policies to verify whether the requested action is permitted. If the policy allows the action, it proceeds; otherwise, it is denied.

For example:

A user requests access to `secret/myapp/credentials`.
Vault checks if the user has a policy that permits `read` access to the `secret/myapp/*` path.
If permitted, Vault returns the secret. If not, the request is denied.

This combination of paths and policies ensures precise, role-based access control, minimizing unauthorized access to sensitive data.

Sequence diagram for Vault authentication and token request.

Best Practices for Using Policies and Paths in Vault

Principle of Least Privilege: Always assign the minimum set of permissions required for users and services. This minimizes the risk of unauthorized access or accidental changes.
Use Granular Policies: Break down your policies by specific paths and actions, ensuring that access is tightly controlled. For example, if a user only needs to read secrets, don’t grant write or delete permissions.
Version Control for Policies: Treat your policies as code, using version control systems like Git. This ensures that any changes to policies are trackable and reversible.
Testing Policies: Before applying policies in production, use Vault’s built-in policy simulation (`vault policy read` or `vault token capabilities`) to test if they work as expected.

Use Cases for Vault

Vault is a versatile tool that can be used in a variety of scenarios to enhance security across an organization’s infrastructure. Some common use cases include:

Secrets Management

The most obvious use case for Vault is the secure storage and management of secrets. Organizations can use Vault to store passwords, API keys, certificates, and other sensitive data, ensuring that only authorized users and applications can access them.

Dynamic Secrets for Databases

Vault can dynamically generate database credentials, which are valid for a short period and automatically revoked after use. This is particularly useful for environments where multiple applications or services need to access a database but you want to minimize the risk of long-lived credentials being compromised.

Encryption as a Service

Vault’s encryption as a service feature allows applications to encrypt and decrypt data without having to manage encryption keys directly. This ensures that data is consistently encrypted using strong keys and reduces the complexity of managing cryptographic operations.

Identity Management and Access Control

Vault can serve as a centralized identity broker, integrating with existing identity providers like LDAP or AWS IAM. This allows organizations to enforce consistent access controls across all applications and services that rely on Vault for secrets management.

Secure Introduction and Authentication

For applications that need to authenticate with external services, Vault can provide secure introduction mechanisms. This allows an application to securely obtain credentials for another service, reducing the risk of credentials being exposed in transit or at rest.

Backend Storage:

Vault needs a storage backend to persist its data. The available backend options include:

Consul (default)
File (for simple setups)
DynamoDB
PostgreSQL

For a production deployment, you would configure Vault with Consul or another backend service that offers high availability and disaster recovery options.

HashiCorp Vault seal and unseal concepts

Sealed State

When Vault is in a "sealed" state, it means that the Vault server is locked and cannot be used to access any secrets or perform any operations. In this state:

The data stored in the Vault’s backend is encrypted, and the encryption keys required to decrypt the data are not available to the server.
Even though the Vault server might be running, it cannot serve requests, retrieve secrets, or perform any operations until it is "unsealed."

Why Does Vault Have a Sealed State?

The sealed state exists for security reasons. The sensitive data in Vault is encrypted with a master key, but that master key itself is encrypted using a separate key derived from the unseal keys. When Vault starts or restarts, it begins in a sealed state to ensure that no secrets can be accessed until authorized users explicitly unseal the Vault.

The sealed state is essentially a protection mechanism, preventing unauthorized access to sensitive data in case the server is compromised during startup or restart.

Unseal Keys

The unseal keys are pieces of a master key used to decrypt the master encryption key of Vault. The unseal process involves providing these unseal keys, which are used to reconstruct the key that Vault uses to decrypt the data stored within it.

When Vault is initialized, a master key is split into multiple unseal key shares using a cryptographic technique known as Shamir’s Secret Sharing. This means that the master key is broken into several parts (unseal keys), and each part is distributed to different authorized individuals.

Key Points about Unseal Keys

Key Shares and Threshold: Vault splits the master key into several unseal key shares (by default, 5 shares), and it requires a certain number of these shares to unseal Vault (typically 3 out of 5). This is referred to as the threshold. For example, if 5 unseal key shares are generated and the threshold is set to 3, you will need at least 3 key holders to provide their unseal keys to unseal Vault.
Partial Knowledge: Each key share is only a part of the master key, so a single key share by itself does not reveal the entire master key or give access to Vault.
Distribution: The unseal keys are typically distributed to different trusted individuals or stored securely, ensuring that no single person can unseal Vault by themselves.

Unsealing Process

To unseal Vault, a certain number of unseal key holders must come together and provide their keys. Vault requires multiple key holders to input their unseal key shares in order to reconstruct the master key and decrypt the secrets stored in the backend.

Here’s how the process works:

After starting, Vault is in a sealed state.
To unseal Vault, an authorized person provides one of the unseal key shares.
Vault will prompt for additional unseal key shares until the required number of shares (threshold) is provided.
Once the threshold is met, Vault reconstructs the master key, decrypts the backend, and transitions to an unsealed (active) state.
Once unsealed, Vault is ready to handle requests and retrieve secrets.

Why Use Unseal Keys and Sealed State?

The combination of the sealed state and unseal keys serves as a robust security mechanism:

Protection Against Compromise: Even if an attacker gains physical access to the Vault server or the data stored in its backend, they cannot access the secrets without unsealing Vault, which requires the unseal keys distributed among multiple trusted individuals.
Fault Tolerance: Splitting the master key using Shamir’s Secret Sharing ensures that no single person has full control over unsealing Vault. The requirement for multiple key holders to cooperate minimizes the risk of accidental or malicious unsealing.
Disaster Recovery: If one of the unseal key holders loses their key, as long as the required number of key shares (threshold) is available, Vault can still be unsealed. This provides redundancy and ensures that the master key can be reconstructed without a single point of failure.

Auto-Unseal Feature (Enterprise)

In the open-source version of Vault, unsealing must be done manually after every server start or restart. However, Vault Enterprise and HCP Vault offer an auto-unseal feature, which automates the unsealing process. Auto-unseal uses a trusted cloud provider’s Key Management Service (KMS) or Hardware Security Module (HSM) to store the master key securely and automate the unseal process. This eliminates the need for manual unsealing, simplifying operations in production environments.

HashiCorp Vault versions

HashiCorp Vault comes in several versions, with different features and licensing options. Here's an overview:

Vault Open Source

Description: This is the free, open-source version of HashiCorp Vault. It includes the core features necessary for managing secrets, such as static secrets storage, dynamic secrets, encryption as a service, and basic authentication methods.
Usage: Vault Open Source is free to use, including in production environments. It's a great option for individuals, small teams, or organizations that need essential secrets management without the need for enterprise features.
Limitations: While robust, Vault Open Source lacks some of the advanced features and enterprise support that larger organizations might require.

Vault Enterprise

Description: Vault Enterprise builds on the open-source version by adding features that are critical for large-scale, production-grade deployments. These include advanced data protection features, performance replication, disaster recovery, namespaces for multi-tenancy, integrated identity management, and more.
Usage: Vault Enterprise is not free. It's designed for organizations that need the extra features for scaling and securing their Vault infrastructure across multiple teams or data centers.
Licensing: This version requires a commercial license. Pricing is typically based on usage and deployment size, and includes support from HashiCorp.

Vault Cloud (HCP Vault)

Description: HashiCorp also offers Vault as a managed service through the HashiCorp Cloud Platform (HCP). This version provides the convenience of having HashiCorp manage the infrastructure, upgrades, and scaling, so you can focus on using Vault rather than maintaining it.
Usage: HCP Vault is also not free. It's a paid service and is ideal for teams that prefer not to manage the complexity of maintaining their own Vault infrastructure.
Licensing: Like Vault Enterprise, HCP Vault requires a commercial license, with pricing based on the scale and usage of the service.

Is Vault Free to Use in Production?

Yes, the Vault Open Source version is free to use in production. It provides a robust set of features for managing secrets, and many organizations start with the open-source version before deciding if they need the additional features available in the Enterprise version.

If your organization requires features like advanced access control, replication, disaster recovery, or enterprise-grade support, you would need to consider the Vault Enterprise version or HCP Vault, both of which come with a licensing cost.

However, for many use cases, the open-source version of Vault is sufficient, and it’s commonly used in production environments without any licensing fees.

Basic Vault Commands

Once Vault is installed and set up, the next step is to initialize and manage secrets using the Vault command-line interface (CLI).

Start Vault

Vault can be started in development mode for testing and experimentation. Development mode is insecure, as it stores data in memory and generates an unseal key and root token upon startup.

To start Vault in development mode, use:

vault server -dev

In production, Vault should be configured with persistent backend storage (such as Consul or a file backend) to store its encrypted data. You'll also want to secure unseal keys and set up secure access policies.

Initializing Vault

Initialization is the first step in setting up Vault. It generates the master key and the root token. The master key is split into several key shares, which are distributed to trusted operators. The root token is used to authenticate with Vault and perform administrative tasks.

vault operator init

The output includes:

Unseal keys: Vault is initialized in a sealed state. To unseal it, you need multiple key holders to provide their unseal key shares.
Root token: This token provides administrative access to Vault and should be carefully guarded.

Unsealing Vault

Vault requires multiple key shares to unseal. This is a security measure to ensure that no single person has the entire master key.

vault operator unseal

Run this command multiple times using different unseal keys (usually 3 out of 5 key shares) until the Vault is fully unsealed.

Logging into Vault

To interact with Vault after unsealing, you need to log in using the root token or a token assigned to a policy.

vault login <root_token>

Once logged in, you can begin interacting with Vault's APIs and commands.

vault login -method=userpass username=username password=test

allows you to authenticate to HashiCorp Vault using the `userpass` authentication method. However, if you're encountering errors like "permission denied" or other issues, there are a few things to check and steps to follow to resolve them.

Loggin with `userpass` correctly

1. Ensure that the `userpass` method is enabled

Before you can use the `userpass` authentication method, you need to make sure it is enabled in Vault. Run the following command to enable the `userpass` method (if it is not already enabled):

vault auth enable userpass

If you get a "permission denied" error, make sure you have the proper permissions (your token must have the permissions to manage authentication methods, as discussed in the previous messages).

2. Verify that the user exists

Make sure the user you're trying to log in with has been created correctly. You can create the user with the following command:

vault write auth/userpass/users/username password="test" policies="default"

Where:

`username` is the name of the user (which must match the one you're using to log in).
`"test"` is the password.
'"default"` is the policy assigned to the user (you can change the policy depending on your needs).

3. Authenticate using `userpass`

After ensuring the `userpass` method is enabled and the user has been created, you can log in with the following command:

vault login -method=userpass username=username password=test

If the login is successful, you'll see a message with a Vault token as a response. This token will allow you to access Vault to perform further operations.

4. Check for any errors

If you encounter errors during login, there could be several reasons:

Incorrect username or password: Make sure the username and password are correct.
Insufficient policy: If the user does not have sufficient policies, you may not be able to log in or perform certain operations after logging in. You can assign more appropriate policies when creating the user.
Authentication method not enabled: If the `userpass` method is not enabled, you will receive an error. In that case, check if you have the permissions to enable the method and proceed with enabling it as shown above.

5. Check Vault's status

If you're still unable to log in and are getting errors, it might be helpful to check Vault's status:

vault status

If Vault is in sealed or standby mode, you won’t be able to authenticate until it is "unsealed."

Storing and Retrieving Secrets

The KV secrets engine

Vault allows you to store secrets in a key-value format. For example, to store a secret:

vault kv put secret/myapp password=mysecretpassword

This stores the `password` key with the value `mysecretpassword` at the path `secret/myapp`.

To retrieve the secret:

vault kv get secret/myapp

The `vault kv put` command is used to store a key-value pair in the KV secrets engine in HashiCorp Vault. Here's a more detailed explanation and an example of how to use the command.

vault kv put <path> <key>=<value>

`vault kv put`: Command to write a secret in the KV store.
`<path>`: The path where you want to store the secret (e.g., `secret/myapp`).
`<key>`: The name of the key for the secret (e.g., `username`).
`<value>`: The value associated with the key (e.g., `admin`).

Suppose you want to store a username and password for an application at the path `secret/myapp`. Here's how to do it:

vault kv put secret/myapp username=admin password=supersecretpassword

In this example:

`secret/myapp` is the path in the KV secret engine.
`username=admin` and `password=supersecretpassword` are the data being stored.

How it Works:

Vault will create a secret at the path `secret/myapp` and store the username and password.
If the path already exists, the command will overwrite the existing data with the new values.

After storing the data, you can retrieve and verify the secret using the `vault kv get` command:

vault kv get secret/myapp

KV Engine Versions

There are two versions of the KV secrets engine:

KV v1: Does not support versioning of secrets.
KV v2: Supports versioning, allowing you to overwrite secrets while keeping track of old versions.

To check which version of the KV engine is in use, you can run the following command:

vault secrets list -detailed

If you're using KV v2, it will appear as `kv-v2` in the `type` column.

KV Engine Versions: A Closer Look

As mentioned earlier, the KV secrets engine comes in two versions:

KV v1:

Stores key-value pairs without versioning.
Overwriting a secret completely replaces the previous one.
Simpler and more lightweight, ideal if versioning isn’t needed.

KV v2:

Adds versioning support, meaning you can store multiple versions of a secret.
You can retrieve or even roll back to a specific version of a secret.
By default, Vault keeps the latest 10 versions of each secret, but this can be configured.

Let's focus more on KV v2 since it's widely used for its flexibility.

Enabling and Configuring KV v2

To enable the KV secrets engine (v2), use the following command:

vault secrets enable -path=secret kv-v2

In this example:

`-path=secret` specifies the path where the secrets will be stored. You can change this to whatever you prefer.
`kv-v2` enables the version 2 of the KV engine.

You can also configure the number of versions Vault keeps with the `max_versions` parameter:

vault write secret/config max_versions=20

This configures Vault to keep up to 20 versions of a secret before purging old ones.

Example of Versioning in KV v2

Let’s walk through a scenario with versioning:

Writing the First Version:

vault kv put secret/myapp username=admin password=supersecretpassword

This creates version 1 of the secret at `secret/myapp`.

Updating the Secret (Creates Version 2):

vault kv put secret/myapp username=admin password=newpassword123

Now, Vault has created version 2 of the secret, keeping track of both the old and new versions.

Retrieving a Specific Version:

You can retrieve specific versions of the secret by specifying the version number:

vault kv get -version=1 secret/myapp

This command retrieves the first version (with the original password).

Listing All Versions:

To see all available versions of a secret:

vault kv metadata get secret/myapp

This will display metadata about the secret, including all version numbers and creation times.

Deleting and Undeleting Secrets in KV v2

In KV v2, you have more flexibility when deleting secrets:

Soft Delete: Vault marks the secret as deleted but retains the version. You can later undelete it.
Permanent Delete: The version is completely removed, and you cannot recover it.

Soft Deleting a Secret Version:

vault kv delete secret/myapp

This deletes the latest version of the secret, but it can still be recovered.

Recovering a Deleted Version:

vault kv undelete -versions=2 secret/myapp

This restores the deleted version (in this case, version 2).

Permanently Deleting a Version:

vault kv metadata delete -versions=2 secret/myapp

This permanently removes version 2 of the secret from Vault.

Deleting Secrets Entirely

If you want to completely remove a secret (including all versions), you can use:

vault kv metadata delete secret/myapp

This command deletes all metadata and versions of the secret.

Policies for Access Control

Vault uses policies to control access to secrets. You can define granular permissions on who can read, write, or delete secrets. For example, you can create a policy like this:

path "secret/data/myapp" { capabilities = ["read", "create", "update"] }

path "secret/metadata/myapp" { capabilities = ["list", "delete"] }

In this policy:

Users can read, create, and update the secrets stored in `secret/data/myapp`.
They can also list and delete the secret metadata (which includes version information).

Example of Policy Usage

To apply the policy, create a file called `myapp-policy.hcl` with the policy content and apply it with:

vault policy write myapp-policy myapp-policy.hcl

Then, you can attach this policy to a user or role to enforce the access control rules.

Practical Example: Using Vault with MFA

Vault can integrate with various authentication methods (including Keycloak). You can require Multi-Factor Authentication (MFA) when accessing secrets by configuring Vault's authentication methods such as OIDC (OpenID Connect), which Keycloak supports.

For example:

Enable OIDC authentication in Vault and configure it to work with Keycloak.
Enforce MFA for critical operations like writing or deleting secrets.

Listing Secrets

To list all secrets stored at a specific path, use:

vault kv list secret

This command lists all stored secrets at the `secret/` path.

Revoking Access

Revoking access to secrets is important when users or services no longer need them. You can revoke leases and tokens.

To revoke a specific lease:

vault lease revoke <lease_id>

To revoke a token:

vault token revoke <token>

Dynamic secret

Dynamic secret management typically includes the following steps:

Request: A client application requests access to a dynamic secret, specifying the role or policy required to access the desired resource.
Authentication and Authorization: Vault authenticates the client and verifies if the client has the necessary permissions to access the requested secret based on assigned policies.
Secret Generation: Upon authorization, Vault dynamically generates a secret. For example, it could be a database credential, API key, or SSH certificate, depending on the backend configured.
Lease Assignment: Vault provides the client with the dynamic secret and associates it with a lease, which defines the secret's lifespan.
Secret Usage: The client uses the secret to access the target resource (e.g., database, cloud provider, etc.) for a limited time.
Lease Renewal (Optional): Before the lease expires, the client can optionally request a lease renewal if it requires extended access. Vault may approve or deny the renewal based on policies and configurations.
Lease Expiry: When the lease expires, the secret is automatically revoked. Vault removes the generated credentials or terminates access, ensuring that the secret is no longer valid.
Secret Revocation (Manual or Automatic): Secrets can be revoked manually by the client or automatically when the lease expires. Vault removes access and cleans up credentials to maintain security.

Dynamic secrets are generated on-demand and automatically expire after a certain period, minimizing

the risk of misuse.

Vault Token and Token accessor

When you run:

vault login -method=userpass username=username password=test

and receive both a token and a token_accessor, here's what you can do with them:

Vault Token (Access Token)

The token is what Vault uses to authenticate you for future requests. After you log in, you can use this token to access various resources within Vault, such as secrets, manage policies, or other services, based on the permissions associated with the token.

Using the token: Once you're authenticated, Vault automatically uses the token for subsequent requests. If you logged in from the command line, the token is cached locally, so you don't need to provide it manually each time.
Inspect the token: You can check the details of your token (such as its lifetime and associated policies) with the following command:

vault token lookup

This command shows information about the current token you're using (expiration, policies attached, etc.).

Manually use the token: If you want to manually use the token for future API requests or other commands, you can explicitly pass it in your requests:

export VAULT_TOKEN=<your-token>

This sets the token in the `VAULT_TOKEN` environment variable, so Vault will use it for every request.

Token expiration: Remember that tokens have an expiration (unless they are "orphan" or "periodic" tokens). When your token expires, you'll need to log in again to get a new one. You can check the expiration time using the `vault token lookup` command.

Token Accessor

The token_accessor is an identifier that allows you to reference the token without exposing the full token value. You can use the accessor for administrative operations, such as revoking the token or managing it securely.

Revoke a token using the token_accessor: Vault administrators can use the `token_accessor` to revoke a token without knowing the actual token value. This is useful for securely revoking tokens without exposing the token itself. For example:

vault token revoke -accessor <token_accessor>

This command revokes the token associated with the provided accessor.

Lookup token information using the token_accessor: You can also retrieve information about a token using the `token_accessor` without exposing the actual token:

vault token lookup -accessor <token_accessor>

This will show you details about the token associated with that accessor (such as policies, remaining lifetime, etc.).

Intermediate and Advanced Vault Commands

Once you're familiar with the basics, you can explore Vault’s advanced capabilities. This section covers enabling secret engines, managing policies, generating tokens, and auditing Vault.

Enabling Secrets Engines

Vault supports several secret engines beyond the key-value store, such as:

Database: To dynamically generate database credentials.
AWS: To dynamically generate AWS IAM credentials.
PKI: To manage certificates.

For example, to enable the AWS secrets engine:

vault secrets enable -path=aws aws

This allows Vault to manage AWS credentials dynamically.

Tuning Vault Policies

Policies define what users or applications can do within Vault. Each policy defines a set of paths and the capabilities allowed at those paths (e.g., `read`, `write`, `list`, `delete`).

Create a policy `myapp-policy.hcl`:

path "secret/myapp"{ capabilities = ["read", "list"]}

Apply the policy:

vault policy write myapp-policy myapp-policy.hcl

This policy grants `read` and `list` capabilities to the `secret/myapp` path.

The command `vault policy write auth-policy auth-policy.hcl` is used to create or update a policy in HashiCorp Vault. In this case:

`vault policy write` is the command to define a new policy or update an existing one.
`auth-policy` is the name of the policy you want to create or update.
auth-policy.hcl` is the file that contains the policy rules in HCL (HashiCorp Configuration Language) format.

This command tells Vault to take the rules specified in the `auth-policy.hcl` file and apply them to a policy named `auth-policy`. This policy can then be associated with specific users, groups, or roles to control access permissions within Vault.

Creating and Managing Tokens

Tokens are used to authenticate with Vault and perform operations. You can create tokens with specific policies:

vault token create -policy=myapp-policy

This creates a new token that is restricted to the permissions defined by the `myapp-policy`.

Auditing Vault

Auditing in Vault tracks all operations performed on the system, which is crucial for security and compliance. Vault supports several audit backends, including file, syslog, and socket.

To enable auditing to a file:

vault audit enable file file_path=/var/log/vault_audit.log

This logs all Vault operations to `/var/log/vault_audit.log`.

Resetting HashiCorp Vault Completely

Sometimes, you might need to reset Vault, either due to misconfigurations, testing purposes, or disaster recovery scenarios. Resetting Vault will remove all stored secrets and configurations, so it must be done carefully.

Step 1: Sealing Vault for Safety

Before resetting, ensure that Vault is sealed so no new operations can be performed:

vault operator seal

This ensures that Vault is in a secure state.

Step 2: Removing Data from Vault

To reset secrets stored in Vault, you can remove them using the `kv` commands:

vault kv metadata delete secret

This command deletes all secrets under the `secret/` path.

Step 3: Deleting Backend Data Storage

Vault's data is stored in a backend (e.g., Consul, file, PostgreSQL). To reset Vault fully, delete the backend storage:

- For file-based storage:

sudo rm -rf /path/to/storage/*

- For Consul:

consul kv delete -recurse vault

Step 4: Vault Server Configuration Reset

If you want to reset Vault’s configuration (e.g., backend settings, policies), delete or modify the Vault configuration file (`vault.hcl`) and remove any old configurations.

Step 5: Re-initializing Vault

Finally, restart the Vault server and initialize it again:

vault operator init

This generates new unseal keys and a root token, effectively resetting Vault to its initial state.

Best Practices for Vault Administration:

Managing HashiCorp Vault requires adhering to key security best practices, particularly in production environments, to maintain the integrity and security of your sensitive data. Here are some essential recommendations:

Follow the Principle of Least Privilege

Always assign the minimum necessary permissions to users and services. This limits potential damage from compromised accounts and ensures users can only access the secrets they require. Start with read-only access for most users and expand permissions only when needed.

Securely Store and Rotate Unseal Keys

Vault’s unseal keys should be distributed securely among trusted individuals or secured in a dedicated hardware security module (HSM). Regularly rotate these keys and ensure they’re not accessible by a single person, as this can prevent unauthorized access to Vault.

Regular Backups of Vault Data

Since Vault stores critical secrets, regularly backing up your Vault data (and the storage backend, such as Consul) is vital for disaster recovery. Be sure to include not only the secrets but also the configuration files and unseal keys.

Minimize Root Token Usage

The root token grants full access to Vault’s administrative functions, making it a high-risk target. Only use it for initial setup or critical tasks, and revoke or limit its usage as much as possible. Delegate other roles and permissions using policies and tokens with more restrictive permissions.

Enable and Monitor Auditing:

Vault provides robust auditing features to log all actions performed on the system, making it easier to track usage and detect suspicious activities. Ensure audit logging is enabled and that you review these logs regularly for any signs of malicious behavior. Integrating Vault’s logs with a centralized logging service can streamline monitoring efforts.

Automate Unsealing with Auto-Unseal:

In high-availability and enterprise environments, manually unsealing Vault after every restart can be cumbersome. Using Vault’s auto-unseal feature, available in the Enterprise version, automates this process by integrating with trusted cloud-based key management services (like AWS KMS) or hardware security modules (HSMs), ensuring Vault remains operational with minimal downtime.

Test Policies in a Staging Environment:

Before applying policies in production, always test them in a staging environment. Use Vault’s built-in policy simulation (`vault policy read` or `vault token capabilities`) to ensure policies work as intended, preventing potential access control issues in production.

Leverage Dynamic Secrets

Wherever possible, use dynamic secrets (e.g., database credentials) instead of static ones. Dynamic secrets reduce the attack surface by generating short-lived, on-demand credentials that are automatically revoked after use.

Conclusion

HashiCorp Vault is an essential tool for securely managing secrets in modern infrastructure environments. By mastering the basic and advanced commands covered in this article, you can ensure that your secrets are managed safely and efficiently. The ability to reset Vault completely is a useful skill for administrators, especially in testing or disaster recovery scenarios.

Vault’s powerful capabilities, from managing dynamic secrets to enforcing fine-grained access controls, make it a vital component in any security-conscious infrastructure. Following best practices, such as securing unseal keys, enabling audit logs, and minimizing root token usage, will ensure that Vault remains a reliable and secure part of your organization’s infrastructure.

Whether you're just starting with Vault or managing a large production environment, understanding these commands and strategies will empower you to use Vault effectively to protect your organization’s sensitive information.

References

HashiCorp. Vault: Documentation.
HashiCorp. Vault - Introduction to Vault.
HashiCorp. Vault Policies.
HashiCorp. Dynamic Secrets in Vault.
Shamir, A. (1979). How to Share a Secret.
Dan McTeer, Bryan Krausen (2020). Running HashiCorp Vault in Production.

Illustrations generated with the help of OpenAI's AI.

Join the Conversation!

I hope this guide has given you a solid foundation for using HashiCorp Vault and its essential commands. But I am just scratching the surface, and I would love to hear from you!

Have you used HashiCorp Vault in your projects?

Do you have any tips, questions, or experiences to share?

Leave a comment below! Whether you're a Vault beginner or an experienced user, your insights can help others on their journey. Let's build a knowledge-sharing space where everyone can learn and grow.

About Me

I am passionate about IT technologies. If you’re interested in learning more or staying updated with my latest articles, feel free to connect with me on:

Feel free to reach out through any of these platforms if you have any questions!

I am excited to hear your thoughts! 👇