chef-vault is the built-in secrets management system for Chef. This post is for people who may have struggled with the documentation and want a simple walkthrough. Once finished with this tutorial, you should be able to implement chef-vault in a compliant way for a security conscious enterprise.

Why chef-vault?

Encrypted data bags force you to copy the shared secret that is used for decryption to your infrastructure. It’s very easy to take that secret file and nefariously decrypt the data from somewhere else without anyone knowing. Chef-vault makes this much more difficult by giving both nodes and Chef server users expressed permission to decrypt certain data. With chef-vault you don’t have to share a secret file with all of your nodes. This is a step up that simplifies everything.

The solution isn’t without its drawbacks. The main one is if you add nodes, you have to rerun something on the server to get that node to be able to decrypt the data bag. With Hashicorp’s vault you get better control over that, and better lease management, and credentials creation. To me, encrypted data bags are like an unreliable used car, chef-vault is a nice mid-size sedan, and Hashicorp’s vault is like a luxury car.

So now that we know where the tool sits within our choices, let’s look at the basics:

Setup

To get started with chef-vault, have the latest Chef Workstation installed and install the chef-vault gem:

chef gem install chef-vault

And then ensure you have a .chef directory that connects to a Chef Server.

Creation

Creating a vault is easy. This creates a vault called passwords:

knife vault create passwords root -S "policy_name:webserver" -A "michael" -J root.json -M client

For whatever reason the knife vault command doesn’t default to talk to a Chef Server. So to create a knife vault, you have to specify -M client at the end which connects to your configured Chef Server. Or you can make your life easier going forward by adding this line to your knife.rb:

knife[:vault_mode] = 'client'

For the command, I used this root.json:

{
  "username": "mhedgpeth",
  "password": "myPassword"
}

This uploads two data bag items to a data bag called passwords:

1. The root data bag item has the data above, encrypted
2. The root_keys data bag item stores the metadata about which clients can read and edit the root data bag item (as you specified above in the search criteria -S and administrators list -A).

Making it Even More Secure

If your onboarding approach isn’t completely locked down, nodes are able to declare their own policy_name and therefore could access these secrets as they join this group.

If this concerns you, specify each node explicitly through the -A flag. So your command would be:

knife vault create passwords root -A "michael,webserver1,webserver2" -J root.json -M client

Viewing a Vault

Now that we have created a vault, let’s view it:

knife vault show passwords root -M client

which will output:

id: root
password: myPassword
username: mhedgpeth

It lets me view it in cleartext because I am one of the administrators on the vault itself. If I want, I can even view it in JSON if you want to move the file to another Chef Server:

knife vault show passwords root -M client -Fjson

Viewing Encrypted Version

To view the encrypted version of the vault, you can simply use the normal commands for viewing data bag, just realizing that the vault data bag also has a _keys item too:

knife data bag show passwords root

and

knife data bag show password root_keys

Will show you lots of encrypted goodness which I will not show. The keys are helpful to see what clients are connected to it.

Adding nodes

Probably the weakest part of chef-vault is what to do when you add nodes. If your nodes grow and shrink dynamically this can be dicey, because when you add nodes, you have to run this command to generate keys for those nodes to read the encrypted data:

knife vault refresh passwords root --clean-unknown-clients

This updates the root_keys encrypted data bag with information on the nodes that now match the search criteria. So it’s important to know that the nodes that can read a vault is a snapshot in time based on the search criteria, not a dynamic list.

If you aren’t using a search criteria, you’ll need to add nodes to the administrators list itself:

knife vault update passwords root -A 'newnode,newnode2'

Rotating keys

You might want to rotate the key that encrypts the data in the data bag. The way this works is the clients use their own key as a private key to combine with the public key on the Chef Server to decrypt the data bag’s key. That key encrypts the real data bag. This command will change that key:

knife vault rotate all keys

Cookbook Development

What use is a data bag without using it in a cookbook? To be able to deal with this data bag in the cookbook, include the chef-vault::default recipe in your runlist. Then you will have the chef_vault_item method that you can call like this:

item = chef_vault_item("passwords", "root")
password = item['password']

Using chef_vault_item will make your cookbook more testable by test kitchen (see below).

Version Control

With data bags, we like to have a data_bags repository that we use to promote shared data and version control changes. This kind of thing doesn’t work with chef-vault. Instead, you get a small team that can update the vault and then have them manually do it. This isn’t ideal, but secrets are hard and, as I wrote above, using a dedicated secrets management tool like Hashicorp Vault will keep you from that level of work.

Kitchen Support

To make this work in kitchen, just put a cleartext data bag in the data_bags folder that your kitchen run refers to (probably in test/integration/data_bags). Then the vault commands fall back into using that dummy data when you use chef_vault_item to retrieve it.

Conclusion

The chef-vault functionality is compelling enough for serious consideration in simple use cases. I would never recommend using encrypted data bags, because the support for chef-vault is more sophisticated without adding a lot of complexity. It’s the right solution for chef secrets when Hashicorp Vault is too complicated or expensive.