Duplicity

Encrypted bandwidth-efficient backup using the rsync algorithm

 

rsync.net supports the use of 'duplicity', which is a compact, elegant tool for automatically encrypting and storing remote backups in a bandwidth efficient manner.

NOTE: If you are running your duplicity backup as root, which is very likely, you should perform all of the commands listed below while actually logged in as root. Do not create your GPG keys, or run the commands shown, under sudo, or you may experience errors such as "encryption failed: public key not found".

 

Installing python and duplicity

 

duplicity is a python script, so python (version 2.3 or higher) must be installed on your system.

In addition, you will need to install GPG and librsync. These items may already be installed on your system...

You may then use your system's package management tool(s) to install duplicity.

OR, you may obtain the duplicity tarball from http://duplicity.nongnu.org/ and after untarring it, install it with:

python setup.py install

You may prefer to install to an alternate target, such as /usr/local:

python setup.py install --prefix=/usr/local

NOTE: You may need to add a --librsync-dir directive to the install command that you run, specifying the exact location of librsync on your system. For instance, on a Mac OSX system using MacPorts, this line may be required to install duplicity:

python setup.py install --librsync-dir=/opt/local

 

Generating SSH keys

 

Although duplicity does have a facility for passing SSH passwords to the remote host (see the --ssh-askpass argument) you probably do not want to be present to type in a password each time your backup runs.

Therefore, if you want to run automated backups, you MUST use passwordless SSH connections (with an SSH key) to use duplicity.

Follow these instructions to set up an SSH key for passwordless logins into your rsync.net account.

 

Generating a GPG Key

 

duplicity encrypts its backups with GPG, which means that you must create at least one GPG key, which duplicity will use for both encrypting the data and signing the encrypted files.

It is possible to create two different keys (possibly owned by different users) and encrypt and sign with different keys, but that is beyond the scope of this document. Please see:

http://www.debian-administration.org/articles/209

for more details.

To create the single GPG key that will be used for both encryption and signing, log in as the user who you will run the backups as, most likely root. The important thing is that it is the same user you used to create your SSH keys above.

Run the command:

gpg --gen-key

The default choices should be chosen. It is up to you how long you want the key to last:

Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits

Once you have created the key, verify it by running this command:

gpg --list-keys

The output should look like:

/root/.gnupg/pubring.gpg
------------------------
pub   1024D/ABAB2AB3 2007-04-23
uid                  Home File 
sub   2048g/12345678 2007-04-23

Two keys are listed - the "pub"lic key and the private key. Make a note of the id of the public key - in this example it is "ABAB2AB3".

 

Running Your Backup

 

It is not advised to place your duplicity backups in the root of your rsync.net filesystem. duplicity generates a lot of files and will quickly become unmanageable unless subdirectories are created.

So, choose a directory in your rsync.net filesystem that you would like to place your duplicity backups in and create it. In this example, we will create a directory called "backups":

ssh 1234@usw-s001.rsync.net mkdir backups

Again, if you run this command as the same user who sent the SSH keys in step #3, you should not be asked for a password. The command will complete without output.

Your first duplicity command will look like:

duplicity full --encrypt-key="ABAB2AB3" /your/files scp://1234@usw-s001.rsync.net/backups

Although this command will generate a lot of output, it should complete without errors. You will be asked for your GPG key passphrase when this backup runs.

If you see errors, most likely beginning with "Traceback (most recent call last):" you should add a -v9 switch to your command:

duplicity full -v9 --encrypt-key="ABAB2AB3" /your/files scp://1234@usw-s001.rsync.net/backups

and send the output to support@rsync.net.

 

Subsequent Runs of duplicity

 

Provided that your first duplicity command ran without errors in step #7, you can simply re-run that same command, using a "inc" command rather than "full":

duplicity inc --encrypt-key="ABAB2AB3" /your/files scp://1234@usw-s001.rsync.net/backups

Each time you run that duplicity will store all the new data that has been produced on your end on your rsync.net filesystem - in the form of diffs against the first backup.

At some point, all of these changes add up - if you continue to run this same command over and over, you will have not only all of your data as it currently exists, but all of your data as it has existed every time you ran it. This is most likely not what you want, and will use a lot of space.

So, decide how many days of backups you would like to retain, and construct a "remove-older-than" command, instructing duplicity to delete backups of your files that are older than X days, etc. For instance:

duplicity remove-older-than 30D -v9 --encrypt-key="ABAB2AB3" --force scp://1234@usw-s001.rsync.net/backups

Please see the duplicity man page for information on the time format of the remove-older-than command.

NOTE: You cannot remove backups that are NEWER than your last full backup. So if you begin performing backups, and updating them with the example shown above, the remove-older-than command will do nothing - since it can't remove data that is newer than the last FULL backup. If you want the remove-older-than command to work properly, you will need to periodically do a FULL backup (the first command shown above) which will allow remove-older-than to remove items that are older than that full backup.

NOTE: duplicity does not allow you to run a backup (full or otherwise) on the same command line as a "remove", or "remove-older-than". Your "remove" directive will need to be run from a separate duplicity command.

NOTE: You will need to add the --force directive anytime deletion or overwrite is required. So you will always need to add the --force directive to any command that uses "remove" or "remove-older-than".

 

Automation

 

First, it should be noted that you can create a GPG keypair, as in the above example, with NO passphrase. This may not be the best practice overall, but if your only goal is to keep your data from residing on rsync.net storage in cleartext, it should suffice, as the passphrase is only for your locally stored private key. So, the below example of exporting your passphrase in a shell script is not necessarily required if you decide to use no GPG passphrase at all. On the other hand, you do need to back up your GPG keys as well, so if you don't have a physically robust method of protecting your keys (like a USB stick in a safe-deposit box) a passphrase may still be required.

It is likely that you do NOT want to type in your GPG passphrase every time you run duplicity. Most likely you want to run duplicity out of your crontab, unattended.

You should create a script that exports your GPG passphrase and then runs the duplicity command:

#!/bin/bash
export PASSPHRASE=yourpassphrase
duplicity inc --encrypt-key "ABAB2AB3" /your/files scp://1234@usw-s001.rsync.net/backups

Make sure to chmod +x the script, and then place that script into your crontab.

 

Verifying and Restoring

 

To verify your backups (we highly recommend that you verify your backups) simply reverse the source and destination in your command and add the verify directive:

duplicity verify --encrypt-key="ABAB2AB3" scp://1234@usw-s001.rsync.net/backups /your/files

This will not restore the files to your end, but will simply compare and verify them. You may wish to add the -v9 switch to your command line to see the full details of the verification.

To restore your data, we run the duplicity command with the source and destination reversed as we did above, with verification. However, as duplicity does not allow restoring into an existing directory, we must create a new directory to restore things in, and reflect that alternate location in our command line:

mkdir /your/restored/files

duplicity --encrypt-key="ABAB2AB3" scp://1234@usw-s001.rsync.net/backups /your/restored/files

As always, all of us at rsync.net strongly encourage you to verify and test the actual restoration of your files on a regular basis, regardless of the method you use to back them up.

 

Additional Notes

 

Hints for Optimization

The current maintainer of duplicity, Kenneth Loafman, offers this useful advice:

I normally keep multiple full backups and my script usually calls out the following in order:

$ duplicity cleanup target --force
$ duplicity remove-older-than 30D target
$ duplicity full|inc source target

By keeping multiple full backups, and by cleaning up and removing the old backups first, I'm able to pack my allocation tightly. One should never have less than two full backups, preferably separated by distance.

I run full backups every Sunday via cron, incremental backups every night. This gives me 4 weeks worth of backup.

 

Backing Up Your GPG Key

The duplicity-encrypted backups that you store will NOT be usable unless you have your gpg secret key. Therefore it is important that you keep a copy of that key somewhere besides the system on which your original data resides on. To export your secret key, run this command:

gpg --export-secret-key [-a] [user name] > keyfilename

and then store the "keyfilename" file that you created somewhere safe, like a CD-ROM in a safe deposit box, etc.

 

Support

 

If you have any questions or problems regarding this process, please email support@rsync.net - we will help you immediately.

 

Reference Information

 

- rsync.net SSH / SSL Server Fingerprints

- Generating and using ssh keys for automated backups

- Remote commands you may run over SSH

- rsync.net Physical delivery guidelines

- rsync.net Warrant Canary

- rsync.net PGP/GPG Public Key

 

Click here for Simple Pricing - Or call 619-819-9156 or email info@rsync.net for more information.

 

 

This duplicity HOWTO and this web page are licensed under a Creative Commons Attribution 4.0 International License.