Since version 4.5.6 (Release Date: 2021-12-17), Kasten by Veeam has supported using the HashiCorp Vault Transit Secrets Engine instead of a passphrase as the Passkey. The Passkey is used by Kasten K10 to protect the master key and encryption key for encrypting snapshots exported to external locations such as S3 or NFS. When using the Vault integration, Kasten K10 sends the ciphertext of the master key to the Transit Secrets Engine and retrieves the decrypted master key. Encryption keys are derived for each policy based on the master key.
For more details on how encryption works in Kasten K10, see my previous post, “Deep Dive: Encryption in Kasten.”
When To Use Vault?
Installing and operating Vault takes quite a bit of effort. If you are already relying on Vault and looking to centralize your secrets management, it’s a good idea to use Vault to protect the Kasten K10 master key. However, if you aren’t already using Vault, we recommend setting it up only for this use case.
If you are running on AWS, you can also use KMS instead of Vault.
Walkthrough: Adding Vault to an Existing Kasten K10 Setup
Installing Vault
Setting up a production-grade Vault cluster is beyond the scope of this blog post, but for demonstration purposes, I’ll set up a single Vault instance in the Kubernetes cluster in which I am running Kasten K10. If you already have a Vault installation, you can skip this step.
If you are still looking into setting up Vault from scratch, my recommendation is:
Note: Interestingly, the Vault Raft Storage Backend uses the same consensus algorithm that etcd uses in Kubernetes. If you want to go full distributed systems engineering nerd, here’s a great strange loop talk you can check out.
Anyway, back to installing Vault. First, we install the Vault helm chart in the new namespace vault:
helm repo add hashicorp <https://helm.releases.hashicorp.com>
kubectl create namespace vault
helm install vault hashicorp/vault --namespace vault
Then we initiate Vault:
kubectl -n vault exec -it vault-0 -- vault operator init
We need to make sure to store the unseal keys and root token!
Finally, we can unseal Vault by providing three unseal keys:
# repeat 3 times until unsealed
kubectl -n vault exec -it vault-0 -- vault operator unseal
kubectl -n vault exec -it vault-0 -- vault operator unseal
kubectl -n vault exec -it vault-0 -- vault operator unseal
Vault is now ready to be used.
We can now view the Vault UI:
kubectl -n vault port-forward vault-0 8200:8200
# Go to: <http://localhost:8200/ui/vault>
# Login using root token provided by the init command
Enabling the “Transit Secrets Engine” in Vault
To use Kasten K10 in Vault, you need to enable the Transit Secrets Engine. “Enable one secret engine” to be precise; in theory, you can have multiple. You can also check out the Vault docs on how to do this.
The following code enables the secret transit engine and mounts it on /transit:
# First login using your root/admin token
kubectl -n vault exec -it vault-0 -- vault login
# Then enable the secret engine
kubectl -n vault exec -it vault-0 -- vault secrets enable transit
Setting Up a Key for Kasten K10
Now that we have the secret engine, we can create a key. This is the key that Kasten K10 will use for protecting its master key:
kubectl -n vault exec -it vault-0 -- vault write -f transit/keys/kasten
Setting Up a Token for Kasten K10
Since we do not want to give Kasten K10 complete access to Vault, we need to create a policy that only allows using the key to encrypt and decrypt data and to view information about the key.
To do so we create the a new file called kasten-policy.hcl :
path "transit/encrypt/kasten" {
capabilities = [ "update" ]
}
path "transit/decrypt/kasten" {
capabilities = [ "update" ]
}
path "transit/keys/kasten" {
capabilities = [ "read" ]
}
Then we use the file to create a policy in Vault:
kubectl -n vault exec -i vault-0 -- vault policy write kasten - < kasten-policy.hcl
Now we can create the token for Kasten K10:
kubectl -n vault exec -it vault-0 -- vault token create -policy=kasten
Write down the token – we will use it in a bit.
Setting Up the Vault Integration in Kasten K10
For Kasten K10 to be able to talk to Vault, we need to set up the integration. First we need to store the token from the previous step in a Kubernetes secret:
kubectl create secret generic vault-creds \
--namespace kasten-io \
--from-literal vault_token=<vault_token>
Next, we need to set the following values in the Kasten helm installation:
- vault.address=http://vault.vault.svc.cluster.local:8200 (if you already have Vault installed, use the URL of your Vault cluster)
- vault.secretName=vault-creds
Depending on how you installed Kasten, you have to take different steps to set the Helm values:
1. If you use a vaules.yaml file, you can add the values in there, then run the helm upgrade command with your values file. It should look something like this:
helm upgrade k10 kasten/k10 --namespace kasten-io -f vaules.yaml
2. If you install Helm without an extra file for the values, use this command:
helm install k10 kasten/k10 --namespace kasten-io \
--set secrets.awsAccessKeyId="${AWS_ACCESS_KEY_ID}" \
--set secrets.awsSecretAccessKey="${AWS_SECRET_ACCESS_KEY}"
Now run the same command but replace the install verb with upgrade and add the two flags at the end:
helm install k10 kasten/k10 --namespace kasten-io \
--set secrets.awsAccessKeyId="${AWS_ACCESS_KEY_ID}" \
--set secrets.awsSecretAccessKey="${AWS_SECRET_ACCESS_KEY}"
--set vault.address=<vault_server_address> \
--set vault.secretName=vault-creds
3. If you used Terraform to manage the Helm release, add the values there:
resource "helm_release" "kasten" {
name = "k10"
# [...]
set {
name = "vault.address"
value = "<http://vault.vault.svc.cluster.local:8200>"
}
set {
name = "vault.secretName"
value = "vault-creds"
}
}
Then run terraform apply again. (In case you’re interested, I also wrote about how to manage Kasten K10 with Terraform previously).
Switching Encryption to Vault
Now that Kasten K10 has access to Vault, we can go ahead and create a new Vault Passkey. This will re-encrypt the master key using the Kasten K10 key in the Vault Transit Engine and store the ciphertext. See the Kasten K10 docs on changing Passkeys for more details.
In the future, Kasten K10 will send the ciphertext of the master key to Vault whenever it needs to access it, for example, when Kasten K10 needs to derive an encryption key for protecting a snapshot that should be exported.
To switch the default Passkey to a Vault Passkey, we need to create a new Passkey by creating the following file passkey-vault.yaml :
apiVersion: vault.kio.kasten.io/v1alpha1
kind: Passkey
metadata:
name: vaultKey
spec:
vaulttransitkeyname: kasten
vaulttransitpath: /transit
usenow: true
We can then create the Passkey using this command:
kubectl create -f passkey-vault.yaml
The new key should now be the only key with in use: true :
kubectl get passkeys.vault.kio.kasten.io -o yaml
Should the creation fail for whatever reason, we can look for more information in the logs of the crypto-svc service:
kubectl -n kasten-io logs -f -l run=crypto-svc -c crypto-svc
Kasten K10 will now use the new Passkey and Vault for encryption. It’s now safe to delete the old key.
Setting Up Kasten K10 with Vault From Scratch
When setting up a fresh Kasten K10 installation, you can also immediately connect Kasten K10 to Vault by creating the vault-creds secret upfront.
Create a special secret with the name k10-cluster-passphrase and use that for the configuration:
kubectl create secret generic k10-cluster-passphrase \
--namespace kasten-io \
--from-literal vaulttransitkeyname=<vault_transit_key_name> \
--from-literal vaulttransitpath=<vault_transit_path>
- You always have the option to configure Vault directly in the Passkey or add a reference to a secret and add the configuration there. However, after you’ve installed Kasten K10, the Passkey API resource already exists, so there is little value in going the extra step to create a secret that the Passkey points to.
- Remember to set the Helm values vault.address and vault.secretName directly on install.
Rotating the Vault Key
The way key rotation works in the Vault Transit Secret Engine is that you can always generate a new version of the key that will be used by default for encrypting plaintext. However, all of the previous versions will still be valid for decrypting ciphertext until you explicitly update the minimally supported version. Vault will choose the right key based on the version string that is embedded in the ciphertext.
The key rotation process has three steps: Create a new version of the Vault key, update all the ciphertext, and update the minimally supported version in Vault.
1. Create the New Key
kubectl -n vault exec -i vault-0 -- vault write -f transit/keys/kasten/rotate
2. Update All of the Ciphertext
To update the Cihphertext, you need to force Kasten to re-encrypt the master key by creating a new Passkey. You can edit passkey-vault.yaml , but rename the key to vaultKey_v2 . It’s good practice to always use the latest version of the key in Vault as the version in the Passkey.
Get the version of the Vault key:
kubectl -n vault exec -i vault-0 -- vault read transit/keys/kasten
Create the new Passkey:
kubectl create -f passkey-vault.yaml
Check that the new key is now also the used key:
kubectl get passkeys.vault.kio.kasten.io vaultKey_v2 -o yaml
# -> Should be "inuse: true"
# Optionally, you can now delete the old key:
kubectl delete passkeys.vault.kio.kasten.io vaultKey
3. Update the Minimally Allowed Version for Decryption
Lastly, it’s important to tell Vault to only decrypt ciphertext using the latest version of the key:
kubectl -n vault exec -i vault-0 -- vault write transit/keys/kasten/config min_decryption_version=2
Automated Key Rotation
I’ve created a script for automating the process above. You can find it on github.com/MoritzKn/kasten-key-rotation
Further Information
%20(1)%20(1).jpg)
