Hashicorp Vault & Jenkins

What is HasiCorp Vault?

It is an open-source tool that helps teams and projects manage and protect sensitive data and secrets. We want to store and use secrets from vault as much as possible to:

Limit secret sprawl
To make it easier to rotate secrets from a central place
To have finer granularity on which pipelines have access to which secrets
Limit secret exposure; If Jenkins server is compromised, the secrets aren’t also compromised
Store the Jenkins Secret backup separately from the main backup.

Setting up a hashicorp vault instance

You can follow Vault’s official documentation to setup a vault instance or use Linode / AWS marketplaces to easi setup a configured instance.

Setup Jenkins access to Vault

We will be using the AppRole feature to enable Jenkins access to Vault. AppRole should generally be used whenever you have a service to connect to Vault.

vault auth enable approle

# Create Jenkins role
vault write auth/approle/role/jenkins-role token_num_uses=0 secret_id_num_uses=0 policies="jenkins"

# Retrieve Approle secrets
vault read auth/approle/role/jenkins-role/role-id
vault write -f auth/approle/role/jenkins-role/secret-id

# Create policy to access secrets
cat << EOF > jenkins-policy.hcl
path "secret/data/jenkins/*" {
 capabilities = ["read", "list"]
}
path "secret/data/jenkins/secret_backup" {
  capabilities = ["create", "read", "update", "patch", "delete", "list"]
}
EOF

# Write policy
vault policy write jenkins jenkins-policy.hcl

Install the “HashiCorp vault” plugin on Jenkins, then navigate to the Jenkins system settings to update the plugin config with the vault server URL and the AppRole credentials with the credential id of ‘jenkins-role’.

Using secrets stored in Vault in Jenkins pipelines

Let’s first create a test secret that we’ll retrieve from a Jenkins pipeline:

# Create KV Store for jenkins
vault secrets enable -path=jenkins kv-v2

# Create test Secret in a path accessible by Jenkins
vault kv put jenkins/test-secret test-secret=123456789

Create a test pipeline to access the secret:

pipeline{
    agent any
    environment { 
        VAULT_URL = '<change to your vault URL>'
        VAULT_CREDENTIAL_ID = 'jenkins-role'
    }
    stages {
        stage('Vault') {
            steps {
                withVault(configuration: [disableChildPoliciesOverride: false, timeout: 60, vaultCredentialId: env.VAULT_CREDENTIAL_ID, vaultUrl: env.VAULT_URL], vaultSecrets: [[path: 'jenkins/test-secret', secretValues: [[envVar: 'TEST_SECRET', vaultKey: 'test-secret']]]]) {
                    sh 'echo "Vault secret is $TEST_SECRET"'
                }
            }
        }
    }
}

To use vault secrets you need to use withVault and provide the path to the secret, they key of the secret you want to extract, and the name of the environment variable to extract the secret to.

Backing up Jenkins secrets directory to Vault

Let’s setup a Jenkins pipeline to write a backup of the secrets directory to Vault every month if anything changes. Create a new pipeline and enable the Build periodically build trigger with the following schedule H 0 1 * * which should equate to once a month.

For Pipeline, choose ‘Pipeline Script’, and paste the following script:

pipeline{
    agent any
    environment { 
        VAULT_URL = '<change to your vault URL>'
        VAULT_CREDENTIAL_ID = 'jenkins-role'
        JENKIS_SECRET_FOLDER = '/var/lib/jenkins/secrets'
    }
    stages {
        stage('Vault') {
            steps {
                withCredentials([[$class: 'VaultTokenCredentialBinding', credentialsId:  env.VAULT_CREDENTIAL_ID, vaultAddr: env.VAULT_URL]]) {

                sh '''#!/bin/bash
                    # Check if we need to update secret
                    touch jenkins_secret_backup_latest_sum_old.txt 
                    sha256sum <(find $JENKIS_SECRET_FOLDER -type f -exec sha256sum {} \\; | sort) > jenkins_secret_backup_latest_sum.txt
                    
                    echo "Old shaSum = $(cat jenkins_secret_backup_latest_sum_old.txt)"
                    echo "New shaSum = $(cat jenkins_secret_backup_latest_sum.txt)"
                    # Skip next steps if hash doesn't change
                    diff jenkins_secret_backup_latest_sum_old.txt jenkins_secret_backup_latest_sum.txt && echo "Files Match" && exit 0
                    echo "Going to backup Jenkins secret folder to Vault"
                    
                    # Compress Jenkins secrets
                    tar -czvf ./jenkins_secret_backup_latest.tar.gz $JENKIS_SECRET_FOLDER 
                    
                    # Create vault secret
                    cat ./jenkins_secret_backup_latest.tar.gz | base64  > jenkins_secret_backup_latest_org.base64
                    tr -d '\n' < jenkins_secret_backup_latest_org.base64 > jenkins_secret_backup_latest.base64
                    
                    cat <<-EOF > jenkins_secret_backup_latest.json
                        { "data": {"secret_dir": "$(cat jenkins_secret_backup_latest.base64 )" }}
EOF
                    #echo  '{ "data": {"secret_dir": "$(cat jenkins_secret_backup_latest.base64 )" }}' > jenkins_secret_backup_latest.json
                    
                    cat jenkins_secret_backup_latest.json
                    
                    # Push secret to vault
                    curl \
                        --header "X-Vault-Token: $VAULT_TOKEN" \
                        --request POST \
                        --data @jenkins_secret_backup_latest.json\
                        $VAULT_ADDR/v1/jenkins/data/secret_backup
                    
                    # Check secret    
                    curl \
                        --header "X-Vault-Token: $VAULT_TOKEN" \
                        --request GET \
                        $VAULT_ADDR/v1/jenkins/data/secret_backup | jq .data.data.secret_dir | tr -d '"'  | tr -d '\n' > jenkins_secret_backup_ret.base64
                    
                    diff jenkins_secret_backup_latest.base64 jenkins_secret_backup_ret.base64 || exit 1
                    
                    # Update hash tracking
                    echo "Updating shaSum of the latest upload"
                    cp jenkins_secret_backup_latest_sum.txt jenkins_secret_backup_latest_sum_old.txt
                    
                '''
                }

            }
        }
    }
}

In a disaster recovery situation, you’ll need to download the backed up secrets directory by using your root or other appropriate vault token, unencode the secret, then copy the tar file to the new jenkins machines and untar to the right directory path.

# Download secret backup
curl \
   --header "X-Vault-Token: $VAULT_TOKEN" \
    --request GET \
    $VAULT_ADDR/v1/jenkins/data/secret_backup | jq .data.data.secret_dir | tr -d '"'  | tr -d '\n' > jenkins_secret_backup_ret.base64

# decode base64 to restore tar
cat jenkins_secret_backup_ret.base64 | base64 --decode >  ./jenkins_secret_backup.tar.gz 

# untar to destination folder
tar -xzvf ./jenkins_secret_backup.tar.gz -C $JENKINS_DST

Hashicorp Vault & Jenkins#

What is HasiCorp Vault?#

Setting up a hashicorp vault instance#

Setup Jenkins access to Vault#

Using secrets stored in Vault in Jenkins pipelines#

Backing up Jenkins secrets directory to Vault#