FreeNAS ZFS snapshot backup to Amazon S3

I’ve been looking for a way to backup my FreeNAS ZFS snapshots to an offsite location. I didn’t find much information how to do this so i had to come up with my own solution.

In this post I’m going to show you how to save your encrypted ZFS snapshots in Amazon S3. We’re going to use a FreeBSD jail together with GnuPG and s3cmd.

Adding a jail in FreeNAS

Go to the FreeNAS web ui and click Jails. Click add and choose name. If you click advanced here you can change the ip-adress for the jail(I wanted to use DHCP).

 

Adding an empty jail in Freenas
Adding an empty jail in FreeNAS

Click ok and FreeNAS will setup a new jail for you which takes a minute or two.

From now on we will have to work in the FreeNAS shell(SSH must be enabled under services in the Web UI).

To list all the jails running on your FreeNAS host we can run:

$ jls

Verify that the jail you created is listed.

To enter the jail run:

$ jexec your_jail_name
$ # Verify that your in the jail
$ hostname
backup

We’re going to need to install some packages. First we need GnuPG which we’ll use to encrypt our snapshots. Then we will need the s3cmd which is used for uploading our snapshots to Amazon s3.

$ pkg install security/gnupg
$ pkg install net/py-s3cmd

I’m going to use symmetric AES256 encryption with a passphrase file because i don’t want to store my data in the cloud unencrypted. So generate a random passphrase which you will need to store at multiple locations(not just inside the jail). Because if the passphrase is lost your backups will be worthless. The passphrase file needs to be accessible by the backupscript. I have placed my passphrase file in in the root directory.

$ echo "mypassphrase" > /root/snapshot-gpg-passphrase
$ chmod 400 /root/snapshot-gpg-passphrase

Next we’ll create a folder holding our current list of snapshots that we’re going to keep synced with S3.

$ mkdir /root/s3_sync_bucket
$ chmod 600 /root/s3_sync_bucket

We also need to configure s3cmd so run this and answer all questions:

$ s3cmd --configure

The backup script

This script should be run on the FreeNAS host. What is does:

  1. Creates a snapshot of the specified dataset
  2. Sends it to the backup jail where it’s encrypted and saved to file
  3. Removes the snapshot on the FreeNAS host
  4. Removes all snapshots older than 7 days
  5. Syncs the local s3 bucket directory with S3 using s3cmd

Create the script and run it manually or with crontab

$ touch /root/backup_script.sh
$ chmod 700 /root/backup_script.sh
$ /root/backup_script.sh my-pool/my-dataset

Edit the script to fit your needs.

Decrypting a backup

To decrypt a backup:

$ gpg --batch --decrypt --passphrase-file /root/pass-gpg < backup_file

ZFS replication from FreeNAS to ubuntu

Recently i’ve have set up an old computer to be used as NAS. I’m using FreeNAS which has some SMB shares and a jail running Nextcloud. The setup is running on a mirror boot device(2x USB drives) with one mirrored volume for data(2x 500GB leftover drives).

FreeNAS is built on top of FreeBSD and ZFS(OpenZFS). When looking at options for backing up the data on FreeNAS i started digging into ZFS with it’s snapshot and replication capabilities. In FreeBSD ZFS comes builtin but with linux because of licens issues you have to install ZFS manually.

Snapshots is a really nice feature of ZFS which is can be seen as point-in-time copy of a dataset. A snapshots diskpace is only the changes in the dataset that has happened which makes it disk space effective. Snapshots can easily be rollbacked, replicated to another machine or mounted at another path.

Of course things got out of hand i’ve struggled for some hours with the ZFS replication to a ubuntu host that i’ve prepared. It turns out that FreeBSD uses a new version of ZFS that makes the receiving side hang with 100% CPU usage as described here #5999. It’s funny because the OpenZFS initiative did a change to use feature flags instead of version numbers to make compatibility less of a problem. Anyhow in 0.7 of ZFS on linux this problem seems to have been fixed. So i tried to compile the sources but got lost somewhere after all those steps required. Then i found this PPA https://launchpad.net/~zfs-native/+archive/ubuntu/daily which only has package for trusty(14.04) so installed trusty and used the precompiled packages.

Installing ZFS on linux(0.7 RC on ubuntu trusty 14.04)

$ sudo add-apt-repository ppa:zfs-native/daily
$ sudo apt-get update
$ sudo apt-get install zfsutils-linux
# Then reboot

Create a simple lab ZFS pool with a file vdev(4GB)

$ dd if=/dev/zero of=example.img bs=1M count=4096
$ sudo zpool create pool-test /home/user/example.img
$ sudo zpool status

Replicate the snapshot

Take a snapshot
zfs snapshot mypool/dataset@snapshotname

To list all snapshots
zfs list -t snapshot

zfs send can be used to send a snapshot to standard output. Send to file with verbose logging.
zfs send -v mypool/dataset@snapshotname > snapshotfile

Send snapshot to another host with ssh(to overwrite existing datataset use -F on the receive command)
zfs send -v mypool/dataset@snapshotname | ssh myhost zfs receive -v myotherpool/newdataset

For the above to work ssh keys needs to be set up between hosts.

Login as root and Set up SSH keys on your FreeNAS host.
ssh-keygen -q -t rsa

Copy the contents of /root/.ssh/id_rsa.pub

On your ubuntu host

# if not already exists
$ mkdir /root/.ssh
# if not already exists
$ touch /root/authorized_keys
# paste the contents of your FreeNAS root id_rsa.pub here
# if not already exists
$ chmod -R 600 /root/.ssh

Then do a manual login from the FreeNAS host to add the ubuntu host to known_hosts.

Conclusion

As of know i probably will use FreeBSD as the receiving side instead or save the snapshots to file and upload to cloud storage.