Manage tenants with Vault namespaces

HashiCorp Vault provides several options for providing applications, teams, or even separate lines of business access to dedicated resources in Vault. For example, you could enable multiple kv (key/value) secret engines using different paths, or use policies to restrict access to specific prefixes within a single secret engine.

An additional method for providing dedicated resources in a Vault deployment are namespaces. All instances of Vault start with the root namespace. With Vault Enterprise and HCP Vault Dedicated, customers can create additional namespaces to allow great isolation of resources. This allows Vault administrators to implement a Vault-as-a-Service model allowing each group using Vault to manage their own secret engines, auth methods, and policies. Most importantly, each group is restricted to work only with the namespace they have been granted access to.

In this tutorial, you will create multiple namespaces, enable plugins, and create policies to demonstrate the isolation capabilities of namespaces.

Prerequisites

• An HCP account with a dev tier or higher cluster.
• Vault installed on your local machine.
• jq installed.

Overview

Namespaces are isolated environments that functionally create "Vaults within a Vault." They have separate login paths, and support creating and managing data isolated to their namespace. This functionality enables you to provide Vault as a service to tenants, allowing each tenant to manage resources within their own namespace.

A tenant, relative to namespaces, is any logical group within your organization that needs to utilize Vault Dedicated. Your Vault design will help dictate how you define a tenant. For example, a tenant may be a specific team such as your front end, or back end development teams. Additionally, you could design Vault around departments, separate lines-of-business, or specific applications where a tenant is your inventory application or marketing automation application.

HCP Vault Dedicated admin namespace

When working with Vault Dedicated, you will start your configuration in the admin namespace. This namespace is created during the deployment of your Vault Dedicated cluster and provides a similar experience to Vault Enterprise clusters that utilize nested namespaces.

Because you are operating from the admin namespace, you must pass the namespace parameter through one of the supported methods such as the -namespace CLI flag, X-Vault-Namespace API header, VAULT_NAMESPACE environment variable, or the vault_namespace resource in Terraform.

Your Vault Dedicated cluster will start with the token auth method enabled. This allows you to log in and perform the initial configuration of Vault Dedicated.

A good practice is to limit the use of the admin token and configure your preferred auth method(s) for regular use. The auth methods you choose to enable will be based on the who (person) or what (machine) will access your Vault Dedicated cluster. It is not uncommon to have multiple auth methods enabled.

For example, you will likely have a team responsible for administering your Vault Dedicated cluster, and applications retrieving secrets. To support human authentication you may choose to enable the Okta, GitHub, or user pass auth method. When an application needs to authenticate with Vault Dedicated, you may choose the AppRole auth method, or a platform specific auth method such as the AWS auth method.

When planning your Vault Dedicated deployment, it is important to consider who or what will need to access Vault, and select one or more auth methods that can support your organizations use case(s).

Vault mount points

Everything in Vault is path-based. Each path corresponds to an operation or secret in Vault, and the Vault API endpoints map to these paths; therefore, writing policies configures the permitted operations to specific secret paths is impacted by the namespace it is created in.

Auth methods and secrets engines can be categorized into two types:

Dedicated: Auth methods and secrets engines that can be managed and mapped directly to a specific organizational unit. For example, the team that manages app-1 can utilize their own AppRole and/or KV mounts without the ability to impact other teams' mounts.

Shared: An organization level resources such as Kubernetes auth method and the Active Directory (AD) secrets engine that are shared and managed at the company or admin level; therefore, mounted at the company or admin level namespace.

You want to find the right balance of granularity between the various mounts needed and the roles defined within the mounts.

Sharing mounts between teams has benefits and risk. It is up to you to find the right balance of granularity between the various mounts needed and the roles defined within the mounts. Below are a couple use cases with their benefits and risks.

Single shared namespace with sub-paths

Benefit: Reduces management overhead from managing multiple namespaces.

Risk: A mount is accidentally deleted causing all users of that secret engine to be impacted.

Namespace per tenant

Benefit: Provide sub-paths for different teams or applications, and limit the blast-radius of an errant change to a single mount.

Risk: Unique mounts per namespace becomes inefficient from a management perspective at scale.

Lab setup

1. Launch the HCP Portal and login.

2. Click Vault in the left navigation pane.

3. In the Vault clusters pane, click vault-cluster.

4. Under Cluster URLs, click Public Cluster URL.

5. In a terminal, set the VAULT_ADDR environment variable to the copied address.

   $ export VAULT_ADDR=<Public_Cluster_URL>

6. Return to the Overview page and click Generate token.

   Within a few moments, a new token will be generated.

7. Copy the Admin Token.

8. Return to the terminal and set the VAULT_TOKEN environment variable.

   $ export VAULT_TOKEN=<token>

9. Set the VAULT_NAMESPACE environment variable to admin.

   $ export VAULT_NAMESPACE=admin

   The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

The Vault Dedicated server is ready.

Personas

This tutorial involves two personas:

• Vault admin (super-user) - responsible for managing the cluster & admin namespace. Uses the Vault Dedicated generated admin token to set up the namespaces, etc.

• Tenant admin - responsible for managing the tenant namespace (e.g. /admin/namespace). Uses an auth method configured by the Vault admin to log in.

Create namespaces

(Persona: Vault admin)

Interactions with Vault Dedicated start in the admin namespace. You can, however, nest namespaces based on your Vault design and organizational needs. In this section you will create two namespaces - namespaceA and namespaceB.

1. Create namespaceA.

   $ vault namespace create namespaceAKey     Value---     -----id      rXVUHpath    admin/namespaceA/

   Notice the path is admin/namespaceA. This is because Vault Dedicated starts with the admin namespace.

2. Create namespaceB.

   $ vault namespace create namespaceBKey     Value---     -----id      rLMNOPpath    admin/namespaceB/

3. List all the available namespaces.

   $ vault namespace listKeys----namespaceA/namespaceB/

   The results display the two namespaces. The partial path is displayed because the admin/ portion of the path was provided to the command by the environment variable VAULT_NAMESPACE.

Enable auth methods

(Persona: Vault admin)

Another design decision to consider when implementing Vault Dedicated is where to enable auth methods. You can enable auth methods in the admin namespace, or you can enable auth methods in a sub-namespace.

The best practice is to enable auth methods in each child namespace and let the namespace-level admin to configure and manage them. This reduces the involvement of the cluster-level admin, and make the child namespace independent from its parent namespace.

You will now enable the userpass auth method in the admin namespace, and namespaceA.

1. Enable the userpass auth method in the admin namespace.

   $ vault auth enable userpassSuccess! Enabled userpass auth method at: userpass/

2. Using the -namespace flag, enable the userpass auth method in namespaceA.

   $ vault auth enable -namespace=admin/namespaceA userpassSuccess! Enabled userpass auth method at: userpass/

   By using the -namespace flag, you have enabled the auth method at admin/namespaceA/userpass.

Create ACL policies

Now that you have created namespaces, you need to create a policy to define what each user can access, and create the individual user accounts.

1. List and review the policies in the admin namespace.

   $ vault policy listdefaulthcp-root

   The hcp-root policy is only available in the admin namespace. This policy provides full access to the admin namespace and any additional namespaces, secret engines, and auth methods you create in Vault Dedicated. You should limit the use of the hcp-root policy. Once you have performed some initial configuration, a good practice would be to create a more limited policy that grants explicit access to the resources needed by your Vault admin and/or security team.

   The default policy is, by default, attached to all authenticated resources, in this case the users you create in Vault Dedicated. This policy provides some basic access to allow the resource to perform token lookups, or access the cubbyhole secrets engine for any individual resource.

2. List and review the policies in the adminA namespace.

   $ vault policy list -namespace=admin/namespaceAdefault

   Each additional namespace you create will also have a default policy available.

3. Create a policy file that allows full access to all paths.

   $ tee vault-admin-policy.hcl <<EOF# Admin policypath "*" {        capabilities = ["sudo","read","create","update","delete","list","patch"]}EOF

   Observe the path is set to "*" meaning any path can be read by the assigned resource.

   What do you think would happen if you assign this policy to a user in the admin namespace?

   What about to a user in namespaceA or namespaceB?

   Would the users in each namespace be able to access all other namespaces?

Test the policy in the admin namespace

1. Create a policy named vault-admin-policy in the admin namespace.

   $ vault policy write vault-admin-policy vault-admin-policy.hclSuccess! Uploaded policy: vault-admin-policy

2. Create a user named vault-admin in the admin namespace and assign the vault-admin-policy policy.

   $ vault write auth/userpass/users/vault-admin \   password=be3xcellent! \   policies=vault-admin-policy \   token_ttl=8h

3. Return to the HCP Portal and click the Public link under Cluster URLs.

4. Click the Method pulldown menu and select Username.

5. Enter vault-admin in the Username field, be3xcellent! in the Password field, and then click Sign In.

6. You are now signed into the Vault UI and placed in admin.

   Based on the policy assigned, should you have full access to the namespace?

7. Click admin and select Manage namespaces.

   You have access to both namespaceA and namespaceB because they are under the scope of the admin namespace, and the policy provides access to all all paths (*).

8. Click the user icon in the top right of the page and select Sign out.

Test the policy in the namespaceA namespace

1. Switch back to the terminal, create a user named vault-admina in the adminA namespace and assign the vault-admin-policy policy.

   $ vault write -namespace=admin/namespaceA auth/userpass/users/vault-admina \   password=be3xcellent! \   policies=vault-admin-policy

2. Read the user details to confirm the user was created.

   $ vault read -namespace=admin/namespaceA auth/userpass/users/vault-adminaKey                        Value---                        -----policies                   [vault-admin-policy]token_bound_cidrs          []token_explicit_max_ttl     0stoken_max_ttl              0stoken_no_default_policy    falsetoken_num_uses             0token_period               0stoken_policies             [vault-admin-policy]token_ttl                  0stoken_type                 default

3. Return to the HCP Portal and click the Public link in the Access web UI card.

4. Click the Method pulldown menu and select Username.

5. In the Namespace textbox enter /namespaceA.

6. Enter vault-admina in the Username field, be3xcellent! in the Password field, and then click Sign In.

7. You are now signed into the Vault UI and placed in namespaceA.

   Based on the policy assigned, should you have full access to the namespace?

   Should you have access to the namespaceB or admin namespace like the vault-admin user has?

8. Click namespaceA and select Manage namespaces.

   You receive a Not authorized message because the policy you created exists in the admin namespace, but the user exists in the namespaceA namespace. The only effective policy is Default.

   Policies must exist in the same namespace as the auth method for the authenticated resource. You cannot assign a policy from namespaceA to a resource in namespaceB.

9. Click the user icon in the top right of the page and select Sign out.

10. Return to the terminal and create a policy in the namespaceA namespace.

    $ vault policy write -namespace=admin/namespaceA vault-admina-policy vault-admin-policy.hclSuccess! Uploaded policy: vault-admina-policy

11. Update the vault-admina user with the new policy.

    $ vault write -namespace=admin/namespaceA auth/userpass/users/vault-admina \   password=be3xcellent! \   policies=vault-admina-policy

12. Return the Vault UI and enter vault-admina in the Username field, be3xcellent! in the Password field, and then click Sign In.

13. Click namespaceA and select Manage namespaces.

    You now have the ability to create additional namespaces under /admin/namespaceA because the vault-admina-policy policy was created in the same namespace as the auth method (userpass) and authenticated resource (vault-admina).

    The policy is relative to the namespace it is created in. The path * allows access to any path within the namespaceA namespace but not the admin or namespaceB namespace.

14. Sign out of the Vault UI.

Test the policy in the namespaceB namespace

Thus far you have created and configured the admin and namespaceA namespace with an auth method, user, and policy. You have logged into each namespace to demonstrate how a policy works relative to the namespace it is created in.

In the admin namespace, using a path of "*" in the policy you were able to access all other namespaces. When you created the a policy in the namespaceA namespace using the same policy file, you were able to only access that namespace and not the admin or namespaceB namespace.

Now, you will simulate creating an admin for namespaceB, but you will create the user in the admin namespace and see how this changes how you may have to write policies.

1. Create a policy file that allows full access to all paths in namespaceB.

   $ tee vault-adminb-policy.hcl <<EOF# Admin policypath "namespaceB/*" {        capabilities = ["sudo","read","create","update","delete","list","patch"]}EOF

   Because the vault-adminb user is being created in the admin namespace, you will need to grant access to the specific mount point for the namespaceB namespace. If you were to reuse the existing vault-admin-policy.hcl file, the user would have access to all namespaces instead of just namespaceB.

   Centralizing all users in the admin namespace is one model that an organization may choose to adopt. It is used to consolidate auth methods to a single namespace and reduce the number of auth methods enabled across all namespaces. A trade off of this mode is that will likely increases the number of policies that are needed because they need to grant or deny permission to specific path.

2. Create a policy named vault-adminb-policy in the admin namespace.

   $ vault policy write vault-adminb-policy vault-adminb-policy.hclSuccess! Uploaded policy: vault-adminb-policy

3. Create user named vault-adminb in the admin namespace and assign the vault-adminb-policy

   $ vault write auth/userpass/users/vault-adminb \   password=be3xcellent! \   policies=vault-adminb-policy

4. Return to the Vault UI.

5. If it is not already selected, click the Method pulldown menu and select Username.

6. Delete any previous namespace text from Namespace text box.

7. Enter vault-adminb in the Username field, be3xcellent! in the Password field, and then click Sign In.

8. You are now signed into the Vault UI and placed in the admin namespace.

9. Click admin and select Manage namespaces.

   You receive a Not authorized message because the policy you created only permits access to namespaceB/*.

10. Click admin and select namespaceB, you are now in the namespaceB namespace.

11. Navigate back to Manage namespaces. Instead of the Not authorized message, you are presented with the Namespaces page where you can create nested namespaces under namespaceB.

12. Sign out of the Vault UI and close the page.

You have successfully created multiple namespaces in Vault Dedicated, enabled the userpass auth method in the admin and namespaceA namespace, and created policies to show how namespaces impact access to resources authenticated to Vault Dedicated.

Using Namespace parameters

Now that you have a working Vault Dedicated environment with multiple namespaces, you can explore the supported options for including the namespace using the Vault CLI and API.

Troubleshooting

Omitting the namespace from any Vault command or API call results in common error messages.

1. Unset the VAULT_NAMESPACE environment variable.

   $ unset VAULT_NAMESPACE

2. Verify there is no VAULT_NAMESPACE environment variable.

   $ printenv | grep VAULT_VAULT_ADDR=https://vault-cluster-public-vault-b0iz2.m3n4l1fe.n1.hashicorp.cloud:8200VAULT_TOKEN=hvs.CAESIBxule_a2c4e6-qhfMZB5TDg8WI8pd8gN0ol0Bhb_5iGicKImh2cy5FcFpCZFJiMUpyeHRkR3hHZTlFenRMR2YudDFsVGQQuAE

   When the VAULT_NAMESPACE environment variable is unset, click the common errors below to produce the example error messages.

$ vault namespace listError listing namespaces: Error making API request.URL: GET https://vault-cluster-public-vault-b0iz2.m3n4l1fe.n1.hashicorp.cloud:8200/v1/sys/namespaces?list=trueCode: 403. Errors:* 1 error occurred:        * permission denied

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \    --request LIST \    $VAULT_ADDR/v1/sys/namespaces | jq -r ".data"

null

Cleanup

To avoid unnecessary charges, you should delete any Vault Dedicated clusters you created for this tutorial.

Help and reference

• HCP Vault Dedicated Overview Documentation
• HCP Vault Dedicated Constraints
• Secure Multi-Tenancy with Namespaces
• Vault Namespace and Mount Structuring Guide