Here's how easy, effective and secure Secrets Management is with HashiCorp Vault

14.09.2022Frederik Möllers
DevOps Authorization / Authentication Certificates CI/CD Cloud Native Computing Foundation DevOps HashiCorp Security

After Ricky has already shown the advantages and necessity of HA key value stores in an older TechUp, this TechUp is about the equivalent for sensitive data. HashiCorp Vault is a tool to store sensitive data like API keys, passwords and certificates not only securely, but also highly available. In the first part of this TechUp we’ll just be looking at the architecture and the basic structure of Vault. Hands-on examples will be in a future article.

Secrets

First of all, the question arises as to what secrets are and why you need your own software to manage them. Secrets are sensitive data to which only a limited group should have access, as they provide access to systems or areas that not everyone is supposed have access to. Examples of this would be passwords, certificates, but also keys for encrypting and decrypting data.

As soon as you move in the context of a larger team, the need to be able to share secrets quickly arises. Starting with the Wifi password, secrets can also be database credentials, for example, which a development team needs for development of its software. A frequently used tool at this point is 1Password, but all major cloud providers also offer a solution for both storing secrets and allowing or restricting access to them. With the need to be able to manage access to a wide variety of secrets in a precise and controlled manner, and to make them available to technical users with high availability, many of these tools reach their limits.

Why HashiCorp Vault?

With Vault, HashiCorp launched a product in 2015 which, as a central element, provides pretty much all use cases and necessities of a larger organization in terms of Secrets and Secret Management. Vault is no longer just a pure secrets store, but a complete secrets management system, which, for example, also offers the option of creating secrets yourself as required. Vault is to be integrated into the key management area of the CNCF landscape.

This is exactly where Vault currently stands out from other tools; As an example, let’s look at the differences to AWS Secrets Manager, the solution from Amazon:

Like Vault, the AWS Secret Manager offers the ability to store, rotate, and manage access to secrets. HashiCorp Vault also offers the option of dynamically creating secrets and deleting them after a defined expiration time. Other features that Vault offers compared to what AWS offers include the ability to generate Certificates, act as a SSH CA Authority, as well as Cross region/Cross Cloud/Cross Datacenter replication and encryption and decryption-as-a-service.

hashicorp-01

Figure: Flow example

This illustration shows an example flow: An application requires database access, the authorizations and configurations were created in advance by an admin in Vault. If an application now needs access to the database, it can query secrets, which are dynamically generated by Vault, from Vault and get access. After a predefined time (TTL) Vault automatically removes the credentials.

Various so-called Secret Engines enable the connection of numerous services to Vault; which function the individual secrets engine has is to be considered for each engine. A secret engine for a database could generate and include secrets dynamically, as in the example above. However, there are also secret engines for simple key-value stores that can store and make available key-value pairs, as well as a secrets engine that decrypts or encrypts data.

Features of Vault

  • Vault handles authentication of clients, validation against third-party trusted sources, as well as authorization and access management
  • Secure encrypted secret storage
  • Dynamic Secrets, which can be generated on-demand for a defined lifetime
  • Encryption and decryption of data outside of Vault
  • Leasing from Secrets
  • Audit Logs
  • Both self-hosted and hosted (HCP Vault)

Architecture

In order to be able to describe the architecture of Vault clearly, after Vault has been installed, we start a Vault server in development mode with the command vault server -dev via the command line:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
vault server -dev
==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.18.5
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.11.2, built 2022-07-29T09:48:47Z
             Version Sha: 3a8aa12eba357ed2de3192b15c99c717afdeb2b5+CHANGES

==> Vault server started! Log data will stream in below:

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: G7zfTKLVRtRJzdWCSPmG3IKFTl3jQY4rpG/WCJsBDUc=
Root Token: hvs.FqHGU7sJXNyVVUhWXhtsASw0

Development mode should NOT be used in production installations!

The server is started in development mode with an instance providing the Vault API on Port 8200. All data is stored in memory (inmem), i.e. not persistently. Also, the dev server starts unsealed which brings us to the first aspect of Vault’s architecture. Before I get into that, let’s set the API address and root token as environment variables to make all further commands more readable:

1
2
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="hvs.FqHGU7sJXNyVVUhWXhtsASw0" 

All data is stored encrypted by Vault in the storage backend. All communication with this backend is encrypted, the key for this encryption and decryption is encrypted in the “sealed” status.

As soon as the Vault Server is started, it is always in the “sealed” state (except if it’s been started as a dev-server) and must therefore first be unsealed before operations are possible. This is possible with an Encryption-Key, which is generated by the Vault Server during initialization and divided into several parts using Shamirs Secret Sharing-Algorithmus, as well as using a Trusted Vault Server, Cloud-Key-Management System or a Hardware Security Module.

In our example, however, the development server only has an unseal-key and is already unsealed. Using the command vault status, we can see the current state of our server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.11.2
Build Date      2022-07-29T09:48:47Z
Storage Type    inmem
Cluster Name    vault-cluster-636af702
Cluster ID      cf682ea5-8497-6052-fb24-cecffad379ac
HA Enabled      false

As we can see, our server is unsealed and has only one secret share. Once the vault is unsealed, HTTP API can be used to communicate and clients can authenticate themselves. Vault offers several options for authentication: Via username and password, public/private keys, tokens, GitHub, cloud services, LDAP, OIDC etc.

After the client has authenticated itself, based on previously created policies a token with a specific lease time is issued, with which the client now has access to the configured areas. Vault only allows access to areas that have been explicitly given via policy. These policies are ACL rules.

In our example, we are currently using the root token to gain access to Vault. If the client now makes a request for a specific secret, it is made available by a so-called Secret Engine and is given a lease ID with which the client can cancel and renew the secret .

Using the command vault secrets list will show us the currently activated secret engines:

1
2
3
4
5
6
7
vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_cc668177    per-token private secret storage
identity/     identity     identity_21538fc7     identity store
secret/       kv           kv_f4a50940           key/value secret storage
sys/          system       system_f2f87035       system endpoints used for control, policy and debugging

On the dev server, the key/value secret engine is activated by default on the path secret/. In the following section, we’ll go into more detail about secret engines.

Secret Engines

As we’ve already seen, Vault offers a variety of options when it comes to authentication. This also applies to secret engines, which are the backend solutions for different types of secrets.

For simple key-value pairs, we have already met the KV Secrets Engine, which can contain both non-versioned and versioned key-value pairs. Another example would be the Kubernetes Secrets Engine, which can generate Kubernetes service account tokens, service accounts, roles and role bindings.

A special feature is the Transit Secret Engine, which can be used to encrypt and decrypt data ‘in transit’ that is not stored directly by Vault. With this cryptography/encription as a service, guaranteed database encryption can be easily implemented on all databases without having to hold the developer responsible.

There is now a secrets engine for almost every use case; whether for all major public cloud providers, Active Directory, databases, as well as for identity management, certificates and SSH. What each Secrets Engine does and which functions it provides can be read in the respective documentation, which HashiCorp is happy to point out:

1
2
3
4
5
6
vault secrets
Usage: vault secrets <subcommand> [options] [args]

  This command groups subcommands for interacting with Vault's secrets engines.
  Each secret engine behaves differently. Please see the documentation for
  more information.

With our key-value secret engine example, secrets can be stored and retrieved as follows: vault kv put -mount=secret hello password=1234

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
vault kv put  -mount=secret hello password=1234
== Secret Path ==
secret/data/hello

======= Metadata =======
Key                Value
---                -----
created_time       2022-08-19T05:31:12.640385Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

The key/value pair we just stored using the put command can be retrieved as follows: vault kv get -mount=secret hello

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
vault kv get -mount=secret hello
== Secret Path ==
secret/data/hello

======= Metadata =======
Key                Value
---                -----
created_time       2022-08-19T05:31:12.640385Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    1234

Because we use the root token, we can store and retrieve secrets without prior configuration. We would first have to give a new client read and write access to the path (in this case secret/) via a policy for it to be able to perform the same actions.

High availability

Since Vault occupies an infrastructure-critical position as a central secret management system, the question of high availability and scaling is obvious.

One way to increase resiliency is to run Vault in HA mode with multiple Vault servers and a storage backend. Here, one of the servers takes on the role of the active server and answers all requests, the other instances are in hot-standby and take on requests if the first one fails.

Since newly started servers are always sealed, as described at the beginning, you should definitely think about one of the “unseal-as-a-service” options in this context. Vault supports various storage backends to persist the data such as Public Cloud Services, Zookeeper, S3, Etcd but also In-Memory and local storage.

Furthermore, since Vault 1.4, a directly integrated storage backend has been available, which is designed for high availability. It uses the RAFT-Consensus-Protocol, which we won’t go into in depth here. This would require several instances for an HA setup. In order to have fault tolerance from a server, at least three HA instances are required. For a two-server tolerance, the cluster would require five instances.

Conclusion

When it comes to Secret Management Systems, HashiCorp Vault is certainly a jack of all trades. Vault consistently focuses on key issues without adding unnecessary complexity. The variety of secret engines and authentication options leave nothing to be desired. Possibilities like encryption-as-a-service top it off. Like all other HashiCorp products, it’s Open Source and self-managed, free of charge. For more information on Vault, I highly recommend the Documentation and the Tutorials provided by HashiCorp.

I hope I was able to give you a good first overview of HashiCorp Vault in this TechUp. In the next part, the focus will be on hands-on. Stay tuned!