This is a text-only version of the following page on https://raymii.org:
---
Title       : 	Basic website and database backups with Duplicity
Author      : 	Remy van Elst
Date        : 	15-02-2013
URL         : 	https://raymii.org/s/tutorials/Website-and-database-backup-with-Duplicity.html
Format      : 	Markdown/HTML
---



This tutorial will show you how you can back up your website and database with
Duplicity. [Duplicity][1] is a very useful tool to make remote unassisted
backups. The backups can be incremental, encrypted, and over a variety of
transports.

<p class="ad"> <b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:</b><br><br> <a href="https://leafnode.nl">I'm developing an open source monitoring app called  Leaf Node Monitoring, for windows, linux & android. Go check it out!</a><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!</a><br><br> </p>


There are a few steps involved in the process of setting this up, and also a few
preconditions:

  * The tutorial works best with a VPS, were you have full root access to the filesystem and database
  * The tutorial is targeted at debian/ubuntu, but with adaption of commands will work under CentOS/RHEL.
  * You need an off-site location to store the backups (other vps for example)

### Basic web application backup

First I'll try to educate you a little bit with some theory.

This tutorial works for all web applications, since they almost all share the
same common structure:

  * source code (.php files, .js files (node), .rb files (rails) etc.)
  * database (MySQL, PostgreSQL, MongoDB etc.)
  * configuration (apache config, nginx config, mysql config, application config)

We are going to back up files. Two of the three things above are already files,
and the database will be exported to a file. The database will be exported so
that it can be imported again using native tools (mysqldump, mongodump etc.),
since just copying the database folders and files will almost always result in a
corrupted database.

Duplicity has a few advantages to other backup tools and scripts:

  * Incremental backups (Saving size)
  * A lot of storage options (ssh, rsync, ftp, amazon S3, IMAP, google drive and more.)
  * Encryption built in
  * Easy to set up and maintain.

I myself have the following set up for almost all of my web application backups:

  * Shell script to export the databases
  * Duplicity backing up the files to a "storage" VPS
  * The storage vps is just a server with a lot of HDD space doing nothing else.
  * Some apps back up daily, some hourly and some weekly.

### Your situation

For this tutorial I'm going to backup a basic Joomla website. Joomla is a
PHP/MySQL CMS. The files are located in `/var/www/joomla`, and the MySQL
database is named `db_054`. I'm also giving the same examples for a Node.js
application using mongodb. The path for that application is `/home/appusr/www/`
and that database is named `uptime`.

You will have to know some things yourself:

  * Where is the application located (`/var/www/joomla`)
  * What database am I using (`MySQL, MongoDB`)
  * How do I export that database (`mysqldump, mongodump`)
  * How do I restore that database (`mongorestore`)

In this example we will use `mysqldump` to backup the MySQL database, and use
duplicity to backup the database dumps, the `/var/www/joomla` folder and the
`/etc/apache2` folder (which has the webserver configuration). If this site ran
over SSL then I would also backup the folder where my SSL certificate was.

We are going to put it all in a cronjob, which runs every 8 hours. We will also
have a full weekly backup, via another cron job.

### Software

Make sure you have the required software installed:

    
    
    sudo apt-get install duplicity gzip python python-paramiko
    

#### MySQL

The mysqldump command to backup the database:

    
    
    mysqldump --single-transaction --routines --events --triggers --add-drop-table --extended-insert -u db_054_u -h 127.0.0.1 -pMyDatabsePassw0rd db_054 | gzip -9 > /var/backups/sql/db_054_$(date +"%H:%M_%d-%m-%Y").sql.gz
    

You will have to replace the `-u db_054_u` with your database user, the
`-pMyDatabasePassw0rd` with your database password and `db_054` with the name of
your database.

The above line will result in a file named like this:
`db_054_15:28_16-02-2013.sql.gz` in the folder `/var/backups/sql`. You will need
to create that folder and make sure you have write permissions to it.

#### MongoDB

For Mongodb I use the `mongodump` command to create a backup of that database:

    
    
    mongodump --host 127.0.0.1:28017 --db uptime --user uptime_u --password uptime_password --out /var/backups/mongo/ 
    

Note that if you want to backup all mongodb databases you can omit the `--db
database` option.

Again replace the corresponding options with your values, and make sure the
`/var/backups/mongo` folder exists and is writable to you.

#### Duplicity

Note: I will not cover the setup of encrypted backups with Duplicity. My backups
go to a trusted server (located at my home). If you need that because you are
backing up to S3, use a search engine to find tutorials on how to set up that.

First we need to create a ssh key to use. We do this with the `ssh-keygen`
program:

    
    
    ~$ ssh-keygen -C 'Duplicity Backup Key'
    
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/remy/.ssh/id_rsa): /home/remy/.ssh/backup_rsa
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /home/remy/.ssh/backup_rsa.
    Your public key has been saved in /home/remy/.ssh/backup_rsa.pub.
    

Make sure you do not enter a passphrase. If you do that, the backup process does
not work unattended.

Now copy over that key to your other (trusted) server:

    
    
    ssh-copy-id -i /home/remy/.ssh/backup_rsa.pub "-p 22 trusted-backup-server.org"
    

And test if you can login without a password with that key:

    
    
    ssh -i /home/remy/.ssh/backup_rsa -p 22 remy@trusted-backup-server.org
    

If that all works continue.

We are now going to set up duplicity. I use the following command to back up all
the folders:

    
    
    duplicity incremental --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --include="/var/backups/sql" --include="/etc/apache2/" --include="/var/www/joomla" --exclude="**" /  ssh://remy@trusted-backup-server.org:22/backups/joomla/
    

If you need to backup more folders, add another `--include="/path/to/folder"`
option. The `--exclude="**" /` option is a trick to backup everything which is
in the include list and nothing else.

For the Node.js/Mongodb application, I would use the following command:

    
    
    duplicity incremental --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --include="/var/backups/mongo" --include="/home/appuser/www/" --exclude="**" /  ssh://remy@trusted-backup-server.org:22/backups/uptime/
    

### Cron

Now putting everything in a cronjob. By using the `;` character after a command,
you can specify multiple commands in one cronjob which run after each other. We
combine the database dump with the duplicity command to set up the back up:

    
    
    10 01 * * 1,2,3,4,5,6 mysqldump --single-transaction --routines --events --triggers --add-drop-table --extended-insert -u db_054_u -h 127.0.0.1 -pMyDatabsePassw0rd db_054 | gzip -9 > /var/backups/sql/db_054_$(date +"%H:%M_%d-%m-%Y").sql.gz; duplicity incremental --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --include="/var/backups/sql" --include="/etc/apache2/" --include="/var/www/joomla" --exclude="**" /  ssh://remy@trusted-backup-server.org:22/backups/joomla/
    

This schedule will create an incremental backup every day of the week at 01:10
AM except sunday. For sunday we have a different cronjob. The only thing
different is the `day of the week` and the duplicity command has the
`incremental` parameter replaced by `full`. This forces duplicity to do a full
backup:

    
    
    10 01 * * 7 mysqldump --single-transaction --routines --events --triggers --add-drop-table --extended-insert -u db_054_u -h 127.0.0.1 -pMyDatabsePassw0rd db_054 | gzip -9 > /var/backups/sql/db_054_$(date +"%H:%M_%d-%m-%Y").sql.gz; duplicity full --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --include="/var/backups/sql" --include="/etc/apache2/" --include="/var/www/joomla" --exclude="**" /  ssh://remy@trusted-backup-server.org:22/backups/joomla/
    

For the node.js/mongodb application we use the following cronjobs:

Incremental:

    
    
    10 01 * * 1,2,3,4,5,6 mongodump --host 127.0.0.1:28017 --db uptime --user uptime_u --password uptime_password --out /var/backups/mongo/; duplicity incremental --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --include="/var/backups/mongo" --include="/home/appuser/www/" --exclude="**" /  ssh://remy@trusted-backup-server.org:22/backups/uptime/
    

Full on sunday:

    
    
    10 01 * * 7 mongodump --host 127.0.0.1:28017 --db uptime --user uptime_u --password uptime_password --out /var/backups/mongo/; duplicity full --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --include="/var/backups/mongo" --include="/home/appuser/www/" --exclude="**" /  ssh://remy@trusted-backup-server.org:22/backups/uptime/
    

### Restoring

Now that we have backups, how do we restore them? Files can just be copied to
the right place, databases need to be imported.

First we get the backups from the backup server:

    
    
    duplicity --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" --file-to-restore / ssh://remy@trusted-backup-server.org:22/backups/joomla/
    

If you need a backup from an earlier date:

    
    
    duplicity --no-encryption --ssh-options="-oProtocol=2 -oIdentityFile=/home/remy/.ssh/backup_rsa" -t 3D --file-to-restore / ssh://remy@trusted-backup-server.org:22/backups/joomla/
    

The `-t 3D` option means restore a backup from three days ago. Things like `-t
1M` (for one month ago) or `-t 5H` (for 5 hours ago) also work.

You will now have the folders and files you backed up. Copy the files back in
place using `cp`. For the databases we use their respective tools.

For MySQL, first `gunzip` the archive, and then import it:

    
    
    gunzip db_054_15:28_16-02-2013.sql.gz
    

And then restore it:

    
    
    mysql -u db_054_u -pMyDatabasePassw0rd db_054 < db_054_15:28_16-02-2013.sql
    

For MongoDB:

    
    
    mongorestore mongo/
    

Make sure you try to restore your backups at least one or twice a month. This
will make sure the backups are usable when you need them!

   [1]: http://duplicity.nongnu.org/
   [2]: https://www.digitalocean.com/?refcode=7435ae6b8212

---

License:
All the text on this website is free as in freedom unless stated otherwise. 
This means you can use it in any way you want, you can copy it, change it 
the way you like and republish it, as long as you release the (modified) 
content under the same license to give others the same freedoms you've got 
and place my name and a link to this site with the article as source.

This site uses Google Analytics for statistics and Google Adwords for 
advertisements. You are tracked and Google knows everything about you. 
Use an adblocker like ublock-origin if you don't want it.

All the code on this website is licensed under the GNU GPL v3 license 
unless already licensed under a license which does not allows this form 
of licensing or if another license is stated on that page / in that software:

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

Just to be clear, the information on this website is for meant for educational 
purposes and you use it at your own risk. I do not take responsibility if you 
screw something up. Use common sense, do not 'rm -rf /' as root for example. 
If you have any questions then do not hesitate to contact me.

See https://raymii.org/s/static/About.html for details.