The general way to send ZFS data sets to remote nodes is normally achieved by sending the ZFS stream through ssh. Example:
zfs send zones/UUID@snapshot | ssh root@10.10.11.5 zfs recv zones/UUID
The down side with this method is that ssh encryption is slow and has significant cpu overhead. You may find yourself in situations where you have very large datasets that you need to move from one ZFS host to another and you want it to be as fast as possible with as little cpu overhead as possible. The following method is good for local environments where you are not concerned about sending streams unencrypted. The tool that works very well for this is “netcat“.
The nc (or netcat) utility is used for a variety of tasks associated with TCP or UDP. nc can open TCP connections, send UDP packets and listen on arbitrary TCP and UDP ports.
The Steps Involved
On the Target SmartOS Node:
We start by running netcat on the target node where we want the ZFS dataset copied to. We run netcat in Listen mode on port 8023 with a timeout of 120 seconds. If we do not specify a timeout then netcat will listen forever and will not exit due to inactivity. This is a bad idea to leave this open on your server indefinitely and you want it to close automatically once the zfs receive is complete. In the last part of the command we pipe whatever data netcat receives into the zfs receive command and specify the target dataset.
nc -w 120 -l -p 8023 | zfs receive zones/UUID
On the Source SmartOS Node:
On the source node we run a ZFS send and pipe it through netcat and specify the target server and port.
zfs send zones/UUID@snapshot | nc -w 20 10.10.11.5 8023
That’s all there is to it!
I am sure you will be pleasantly surprised by the speeds you achieve as well as the significantly lower cpu overhead.
Using mbuffer to speed up slow zfs send | zfs receive
By Alasdair Lumsden on 18 Jul 2010
So, you find yourself doing a zfs send | receive, perhaps a large incremental send. You find the transfer process going really slowly – trickling along at less than 1MB/sec. Yet, you know the sender and the receiver is capable of far far more than this. What’s the deal?
Well, basically, zfs receive is bursty – it can spend ages computing something, doing no receiving, then blat the data out to disk. The issue with this is that it stalls the sender, resulting in a bursty and slow transfer process.
The solution is to deploy mbuffer into the mix. MBuffer will buffer the stream, which you can do at both ends. While ZFS receive can’t receive, mbuffer buffers, when zfs receive can receive, mbuffer sends it data as fast as it can. Let’s see an example:
# Start the receiver first. This listens on port 9090, has a 1GB buffer,
and uses 128kb chunks (same as zfs):
mbuffer -s 128k -m 1G -I 9090 | zfs receive data/filesystem
# Now we send the data, also sending it through mbuffer:
zfs send -i data/filesystem@1 data/filesystem@2 | mbuffer -s
128k -m 1G -O 10.0.0.1:9090
You’ll get a lovely output such as:
in @ 15.8 MB/s, out @ 8923 kB/s, 1.0 GB total, buffer 6% full
And hopefully, you’ll find your zfs send|receive suddenly go a lot lot quicker.
Here is the link to download the useful mbuffer program.
I personally found this approach decreased my send time for 40GB from over 4 hours down to around 30 minutes. Hurrah! 🙂