Overall

ZFS replication works with snapshots. First, create a base snapshot, then initiate send/recv to send that over to the target system. When it’s received, the target system will be populated with the zfs data, along with the snapshot. To perform an incremental snapshot, create another snapshot on the source, then use the incremental flag and zfs send will push the delta between the two snapshots to the target.

I’ve prepared a screencast demonstration here http://youtu.be/TwsqKBrZ1t8

Test system

I’m running this on Solaris 11 x86. My zfs volume is called «rpool/testrep». The source system is 10.0.88.88 and the target is 10.0.88.89.

Create a snapshot

# zfs snapshot -r rpool/testrep@s1

Send the filesystem to the remote server

I’m using ssh key authentication but it should also work with password.

# zfs send rpool/testrep@s1 | ssh 10.0.88.89 zfs recv -F rpool/testrep

Making a second snapshot

Now make some changes to rpool/testrep and create another snapshot called «s2»

# zfs snapshot -r rpool/testrep@s2
# zfs list -t snapshot
NAME                             USED  AVAIL  REFER  MOUNTPOINT
rpool/ROOT/solaris@install      7.86M      -  1.92G  -
rpool/ROOT/solaris/var@install  1.72M      -  97.0M  -
rpool/testrep@s1                20.5K      -   103K  -
rpool/testrep@s2                  21K      -   134K  -

Making an incremental replication

zfs send -R -i rpool/testrep@s1 rpool/testrep@s2 | ssh 10.0.88.89 zfs recv rpool/testrep

What if the target server has dirty data?

In some case, the target server may receive file changes. ZFS will refuse to replicate if the target server has its files modified.

# zfs send -R -i rpool/testrep@s3 rpool/testrep@s4 | ssh 10.0.88.89 zfs recv rpool/testrep
cannot receive incremental stream: destination rpool/testrep has been modified
since most recent snapshot

Easy enough, just rollback the target system to a snapshot of your choice

zfs rollback rpool/testrep@s3

And then do an incremental replication to bring it to s4.

Clean up

When you no longer need the snapshot, simply destroy them one by one.

zfs destroy rpool/testrep@s4

Monitor throughput

zpool iostat 1