The Re-test

I’ve gradually been storing more and more data on there, and while I’ve happily recovered the odd file or two from it, it had been a while since I tested a full-sized recovery job.

To re-test this I decided to download a fairly typical development folder, with around 20,000 mostly quite small files in it.

The Problem

This is where I discovered a problem. I started the recovery one evening when I’d finished for the night, and left the system busily restoring overnight.

I returned the following morning fully expected everything to be sat there waiting for me. I was therefore rather surprised to discover that only about 12% of the files had been restored.

Finding An Alternative

So I decided that I had to find a better faster alternative.

After a lot of research, I decided that OneDrive looked like a good option. It natively integrates with Windows and while (unlike with SpiderOak) the data stored on OneDrive isn’t encrypted at rest, Microsoft does seem to take data privacy and protection seriously which I feel is fine for most of the personal documents I have.

So I simply added these to the OneDrive folder and bingo, the syncing began and has worked pretty much flawlessly since.

However, I’m not happy to leave either client data or server backups unencrypted at rest. This meant I had to find yet another option, preferably one that worked with OneDrive.

Encrypting Important Data

After doing the research, I found the perfect solution. It’s easy to implement, utilises the OneDrive datastore and encrypts the entire backup even at rest. It’s also lightning fast to restore an entire backup, not just one directory of 20,000 files. Perfect!

Note: while Microsoft does offer an encrypted vault facility, it’s not really practical to use regularly at this time as it automatically locks every few minutes which can become very frustrating!

Restic Backup With Rclone And OneDrive

The following guide will show you how to install Restic and Rclone, configure Rclone to connect to OneDrive and create a simple backup script to backup MySQL databases and other files to OneDrive that can be used with cron for regularly scheduled backups.

Part of this process needs you to connect rclone to OneDrive and create a remote (connection) for it to use. The easiest way to do this requires that you can access your localhost with a web browser but you can also do this on a machine with no Internet browser available.

In this guide, I have used Windows 10 with its Subsystem for Linux (Debian) but the basic steps should work for any suitable platform.

Let’s Begin

To get started, open up the WSL command prompt and create a new folder for your backup files and scripts.

mkdir backup
cd backup

All of the steps below will assume that this is your current working directory and you have root-level access.

Now, you will need to install the latest version of rclone and restic backup.

Installing rclone

I strongly recommend installing this directly from the rclone website instead of your Linux distro’s repository. This is just to ensure you get the latest version.

curl https://rclone.org/install.sh | sudo bash

Installing restic

Again, I recommend downloading the precompiled binary from the restic website instead of using your Linux distro’s repository.

Once downloaded, extract and copy the executable file to the WSL folder you created above.

chmod 755 restic

Tip: you can access the root WSL file system from Windows Explorer by entering \\wsl$ in Explorer’s address bar.

Now rename the executable to ‘restic’ and make sure the file is executable.

cd backup
mv restic_0.9.6_linux_amd64 restic
chmod 755 restic

Connect To OneDrive

Now you can connect rclone to OneDrive and create a connection (remote) for the backup.

rclone config

This will start the rclone interactive configuration dialog that will walk you through configuring the remote.

First, you need to choose an option from the list of options. Select ‘New Remote’ by entering ‘n’.

It will then prompt you for a name for the new remote. In this example, I will use ‘backup’ but it could be anything of your choosing. You will use this remote name in later steps.

Next rclone will ask for the type of storage you want to configure and display a list of available options. You will need to find the ‘Microsoft OneDrive’ option and enter that number. At the time of writing, this was ’23’.

After that it will ask for your ‘Microsoft App Client ID’ followed by your ‘Microsoft App Client Secret’. You can normally just accept the (blank) default for these.

The next question is whether you want to ‘Edit The Advanced Setup’. Answer ‘n’ to this and continue to the next step.

Now it will ask if you want to ‘Use Auto Config’. This is where you will need an internet browser available to access the localhost (127.0.0.1). Answering ‘n’ to this will cause rclone to display a URL something like:

http://127.0.0.1:53682/auth?state=xxxxxxxxxxxxxxxxxxxxxxx

At this point rclone will pause and be ‘Waiting For Auth Code’. To get the code, you need to copy the link and paste it into your browser. This will take you to Microsoft’s website where you can log in with your OneDrive account.

Once you’ve completed this step, rclone will automatically receive the authorisation code and continue to the next step.

You now need to confirm your account type from the list rclone will display. You need to choose the ‘OneDrive Personal Or Business’ option. At the time of writing, this was option ‘1’.

After confirming this, rclone will display a list of available drives. In most cases, there will only be one listed, and you can select option ‘0’ (zero). Confirm this is OK by entering ‘y’.

Finally, rclone will display the configuration details. Confirm this is also OK by entering ‘y’ again.

This will take you back to the initial options menu. As you have now completed the configuration, you can select the ‘Quit Config’ option here by entering ‘q’.

Testing Your Rclone Configuration

The easiest way to test the configuration is to print a remote file listing. You can do this by using the ls command:

rclone ls backup:

Where ‘backup’ is the name that you entered earlier for the rclone remote. Don’t forget the colon (:) after the name!

This will print a list of any files you already have stored on OneDrive. If you want to see a list of directories (folders) instead, use ‘lsd’ instead of ‘ls’.

Restic Backup

OK, so if you’ve got this far you must have installed rclone and created a working remote. Well done!

The next thing to do is to create a new repository for the restic backup. You can do this by entering the command below:

./restic -r rclone:<remote name>:<remote path> init

The <remote name> is the name of the rclone remote you created above, and the <remote path> is the path on OneDrive where you want the repository files to be stored. So for this example, I could use:

./restic -r rclone:backups:mybackupfolder/mypc init

If the remote folder doesn’t exist, it will be created for you.

The first time you run this, you will be prompted for a backup password. This will be used to encrypt the remote backup.

Before you can start backing up, you will now need to create a password file that contains a single line with your chosen password. You can do this using your favourite editor, for example:

vi backup-pwd

You can name this file anything you like, but for the purposes of this article and the script below, I have called it ‘backup-pwd’.

Once you’ve created the password file, you can start backing up using the command below:

./restic -p <password file> -r rclone:<remote name>:<remote folder> backup <folder to backup>

So as an example:

./restic -p "backup-pwd" -r rclone:backup:mybackupfolder/mypc backup "/mnt/c/Users/Steve/Documents"

This would backup everything from my Documents folder on my local C: drive by creating what restic calls a snapshot. If I run the same command a second time, then restic will create a new snapshot and backup any changes.

You can also backup a list of folders by creating a simple text file that contains a list of the folders. For example:

/mnt/c/Users/Steve/Documents
/mnt/c/Users/Steve/Picture

You can save this under any name you like, but in this article and the script below I’ve used the file name of ‘backup-file-list’.

To backup using this file, change the backup command to use the –files-from option instead of the <folder to backup>. As an example:

./restic -p "backup-pwd"-r rclone:backup:backup/dev backup --files-from "backup-file-list"

This would backup both my Documents and Pictures folders from my local C: drive.

View Snapshots

Use the following command to view existing snapshots:

./restic -p backup-pwd -r rclone:backup:mybackupfolder/mypc snapshots

Restore Snapshots

Use the following command to restore snapshots:

./restic -p "backup-pwd" -r rclone:backup:mybackupfolder/mypc restore latest --target <destination folder>

Where <destination folder> is the local folder you would like the backup restored into. So for example:

./restic -p "backup-pwd" -r rclone:backup:mybackupfolder/mypc restore latest --target ./myrestoredbackup

Scheduling A Backup

You can take things a step further by creating a backup shell script in the WSL system and adding it as a Windows Task Scheduler job. You can also include database and crontab backups in this script, or anything else you need. For example:

bash ~/scripts/backup.sh

The example bash script below might be useful as a starting point:

#!/bin/bash

# setup config
# where to put the db backups
BK_DEST="/root/data"
# mysql database backup login
BK_MYSQLSERVER="localhost"
BK_MYSQLUSER="<backup user>"
BK_MYSQLPASSWORD="<backup user password>"

# array of mysql databases to backup
BK_MYSQLDBS=( "db1" "db2" "etc" )
# if no databases 
# BK_MYSQLDBS=( )
echo "Starting backup"

# prepare for backup and create db backup folder if it doesn't already exist
if [ ! -d "$BK_DEST" ]; then
  # check that list of databases is not empty
  if [ -n "$BK_MYSQLDBS" ]; then
    echo " Creating local backup folder $BK_DEST"
    if ! mkdir -p $BK_DEST; then
      echo " PROBLEM CREATING BACKUP FOLDER"
      exit 1;
    fi
    echo " Creating backup folders"
    mkdir $BK_DEST/mysql
    mkdir $BK_DEST/crontabs
  fi
fi

# backup mysql databases listed in BK_MYSQLDBS
echo "Backing up databases"
for BK_DB in ${BK_MYSQLDBS[@]}; do
  echo " Processing $BK_DB"
  if ! eval "mysqldump -u$BK_MYSQLUSER -p$BK_MYSQLPASSWORD --triggers --routines --single-transaction --flush-logs $BK_DB |gzip -c > $BK_DEST/mysql/$BK_DB.sql.gz"; then
    echo " PROBLEM DUMPING DATABASE"
  fi
done

echo "Running Off-site Backup"
echo " -> OneDrive"
./restic -p "backup-pwd" -r rclone:backup:mybackupfolder/mypc backup --files-from "backup-file-list"
echo "Done"

You will notice that the MySQL database backup just overwrites the backup file each time. This is because the restic snapshots take care of the versioning and store progressive incremental backups on the OneDrive folder until you delete them.

Tip: Remember also, you can also create a log by using the Linux >> operator at the end of the scheduler action to append the script output to a log file. For example:

bash ~/scripts/backup.sh >> ~/scripts/backup.log

Other Useful Commands

Remove A Single Snapshot

restic -p backup-pwd -r rclone:backup:mybackupfolder/mypc forget <snapshot id>

Remove older snapshots

For example, delete all but the last 10 snapshots:

restic -p backup-pwd -r rclone:backup:mybackupfolder/mypc forget --keep-last 10

See Also

Rclone project

Restic backup project

Configuring rclone for OneDrive

SpiderOAK Server Backup

Found This Useful?

If you have found this useful, why not help to support the site and buy me a coffee or perhaps a cheeky beer? Thanks!

Websites Built For You

You may also like Websites Built For You which focuses on web design and development in WordPress, PHP and Javascript.

The post OneDrive Server Backup appeared first on perkinstuff.com.

The script assembles files, exports databases and crontabs and compresses everything into a single gzipped tar archive. It then encrypts the archive and uploads the encrypted data via ncftp to an offsite backup host.

This works great for a while but as the data grows so does the size of the backup.

As an aside to this if anyone is looking for a great FTP backup service I can thoroughly recommend ADrive as being very reliable and excellent value for money.

Simple Incremental Encrypted Backups

As they do, servers tend to grow and the one big file approach ends up with bigger and bigger backup archives that fail to upload more and more frequently.

So I started looking for a better way. For me it had to be 1) simple 2) incremental 3) encrypted and 4) (if possible) have a nice desktop client that I could access it through.

I’d recently started using SpiderOAK as a replacement for Dropbox due to privacy concerns. While searching for a simple server backup solution I came across their command line documentation.

So I thought I’d give it a go. Bingo! It was the server backup solution I’d been looking for.

So here it is my quick guide to setting up simple server backup using the SpiderOakONE command line utility on the Debian Linux operating system.

Installation

This guide assumes you already have a SpiderOakONE account configured and ready to go.

The client is easy to install, you just need to add in the SpiderOak repository, their gpg key and apt-get away:

echo 'deb http://apt.spideroak.com/debian/ stable non-free' >> /etc/apt/sources.list
gpg --keyserver pgpkeys.mit.edu --recv-key 573E3D1C51AE1B3D
gpg -a --export 573E3D1C51AE1B3D | sudo apt-key add -
apt-get update
apt-get install spideroakone

Now that you have the client installed, you need to connect it to your SpiderOak account. Enter the following and follow the on-screen prompts:

SpiderOakONE --setup=-

Once you’ve entered the requested information, it will start to synchronise with the SpiderOak servers. This can take a few minutes so probably a good time to get yourself a coffee!

Talking about coffee, if you find this or any of the other posts on my site helpful, perhaps you’d like to (shameless plug coming up) buy me one as well! There’s a link at the bottom of the post.

Running A Backup

Ok, so now that you have everything installed and ready to go, how do you run a backup. The simplest way is via the comamnd line, so for example:

SpiderOakONE --backup=/var/www

Would back up the contents of /var/www directory to SpiderOak’s encrypted backup servers.

Scheduling A Backup

You can take things a step further by creating a backup script and scheduling it as a cron job. You can also include database and crontab backups in this script, or anything else you need. The example bash script below might be useful as a starting point:

#!/bin/bash

# setup config
# where to put the db backups
BK_DEST="/root/data"
# mysql database backup login
BK_MYSQLSERVER="localhost"
BK_MYSQLUSER="<backup user>"
BK_MYSQLPASSWORD="<backup user password>"

# array of mysql databases to backup
BK_MYSQLDBS=( "db1" "db2" "etc" )
# if no databases 
# BK_MYSQLDBS=( )
echo "Starting backup"

# prepare for backup and create db backup folder if it doesn't already exist
if [ ! -d "$BK_DEST" ]; then
  # check that list of databases is not empty
  if [ -n "$BK_MYSQLDBS" ]; then
    echo " Creating local backup folder $BK_DEST"
    if ! mkdir -p $BK_DEST; then
      echo " PROBLEM CREATING BACKUP FOLDER"
      exit 1;
    fi
    echo " Creating backup folders"
    mkdir $BK_DEST/mysql
    mkdir $BK_DEST/crontabs
  fi
fi

# backup mysql databases listed in BK_MYSQLDBS
echo "Backing up databases"
for BK_DB in ${BK_MYSQLDBS[@]}; do
  echo " Processing $BK_DB"
  if ! eval "mysqldump -u$BK_MYSQLUSER -p$BK_MYSQLPASSWORD --triggers --routines --single-transaction --flush-logs $BK_DB |gzip -c > $BK_DEST/mysql/$BK_DB.sql.gz"; then
    echo " PROBLEM DUMPING DATABASE"
  fi
done

# spideroak
SpiderOakONE --backup=$BK_DEST
SpiderOakONE --backup=/var/www
SpiderOakONE --backup=/etc

You will notice that the MySQL database backup just overwrites the backup file each time. This is because the SpiderOak client takes care of the versioning and stores progressive incremental backups on their servers until you delete them.

Email Notifications

You can set up cron to email you the results of the backup. Nullmailer is my favourite and you can find more information that here.

The SpiderOak client also comes with all kinds of advanced backup commands including their own scheduling system but those are way beyond the scope of this article. Keeping it simple works for me, but you can find more by visiting the SpiderOak Command Line Reference documention.

Thanks for reading!

Found This Useful?

If you have found this useful, why not help to support the site and buy me a coffee or perhaps a cheeky beer? Thanks!

Websites Built For You

You may also like Websites Built For You which focuses on web design and development in WordPress, PHP and Javascript.

The post SpiderOAK Server Backup appeared first on perkinstuff.com.