Backing up Jenkins

Now that I have a running Jenkins instance with a few pipelines setup, I would like to be able to easily recover my setup in case something happens to the main instance. Also, it’s good practice to regularly backup things like your configurations, Jenkins, and databases. But, what do you need to back up exactly?

What to Backup

It depends exactly on your specific use-case and needs. Do you need to restore the exact version of plugins you had installed? Do you need access to the builds history? Do you need access to old build artifacts? Logs? Jenkins has a guide that details the minimal amount you need to backup in order to restore your pipelines which is a good start. You can add more directories to backup, like jobs, workspaces, plugins, etc.. as the need comes up. I recommend taking time to truly think of your needs and only include what’s truly necessary.

Speaking of build results and artifacts, make sure you have configured Jenkins to delete old results and artifacts after X days or some other rule to avoid using up all the disk space as well as minimizing the amount of data that will be backed up, if you choose to back up the builds.

Performing the Backups

You can use a plugin such as thinBackup to create regular backups or a bash script run by a cron job to do the same. I like to setup a schedule of once a week, 0 0 * * /1 * , or every two weeks (14th and 28th of the month) 0 0 */14,28 * *

Where to store the backup?

The backups can be stored in your storage solution of choice. In my case, I want to store them in a versioned S3 bucket.

# Assuming ThinBackup was configured to use `~/jenkins_backup` as backup directory

# Archive backup 
cd ~/jenkins_backup
ls -Art  | tail -n 1 |  xargs tar -czvf ../jenkins_backup_latest.tar.gz

# Push to S3 
aws s3 cp jenkins_backup_latest.tar.gz s3://{bucket_name}

You can setup a cron job to upload the back up S3 or setup a Jenkins pipeline to do that.

Where to store the secrets?

The secrets directory at $JENKINS_HOME should be backed up to a different place than the main jenkins backup. Could be an external disk, a cloud secret manager, such as AWS KMS, or using a third-party solution such as Hashicorp Vault.

I will explore using Hashicorp Vault to backup Jenkins installation secrets and how to use secrets stored in Vault within your Jenkins pipeline in a future post.

Restoring backups:

# Restore Jenkins Configuration
## Download from S3
aws s3 cp s3://{bucket_name}/jenkins_backup_latest.tar.gz jenkins_backup_latest.tar.gz

## Restore
export JENKINS_DST="/home/jenkins/jenkins_backup_test" # or Set to $JENKINS_HOME if you're performing a real recovery 
tar -xzvf jenkins_backup_latest.tar.gz -C $JENKINS_DST

# Restore Jenkins secret keys 
export JENKINS_SECRETS="/home/jenkins/jenkins_backup_secrets"
export JENKINS_HOME="$JENKINS_DST/$(ls -Art  jenkins_backup_test | tail -n 1)"
## Download 
### Move secret from KMS, external storage, vault, etc.. to jenkins machine
## Copy over 
cp -r $JENKINS_SECRETS  $JENKINS_HOME/secrets

# Test Backup 
export JENKINS_HOME="$JENKINS_DST/$(ls -Art  jenkins_backup_test | tail -n 1)"
java -jar /usr/share/java/jenkins.war --httpPort=9999
# NOTE: If you didn't backup the plugins, you will need to reinstall the plugins then restart the jenkins server to see your jobs and credentials again.

If you use ThinBackup, you can use it to restore a specific backup version from the backups directory.