In today’s post, I’m sharing an interesting article I came across recently.

Most of DIY readership will be familiar with rsync.  It’s a fantastic technology and tool for making a directory in one place exactly like a directory on another.  You can use it to synchronize two directories on a server, or two directories on different servers (either over ssh or via the rsync server, though the former is probably more common).

One of the huge benefits of rsync is that it only copies changes.  So if you have a 100TB filesystem but only 1GB has changed, only that 1GB will be copied.  In fact, if you have a 10GB file and only 1GB of that file has changed, only 1GB will be copied!  This can be a huge time savings, particularly if the copying medium is something constrained, like network bandwidth.

However, rsync has one really key drawback: it processes changes serially.  This isn’t a big deal if you have a directory with 100 files.  It’s a huge deal if you’ve got a filesystem with 100,000 files.  rsync is going to go through each file one by one, determining if there are changes, and then transmitting those changes.

Is there a better way?  Yes!  rclone.

rclone is a great tool that is sort of “rsync on steroids”.  It supports backing up (rsync-like) to a wide variety of cloud services.  If you have a filesystem you want to backup, you can use rclone to sync it to Google Drive, Dropbox, B2, an SFTP site, and tons more.

You can also target another filesystem on the same host.  And the cool thing about rclone is that it can run with multiple threads.

When I use rclone to backup some of my filesystems to cloud storage for backup, I use 16 threads in parallel.  That gives it a very fast metadata scan and full utilization of the available bandwidth.  Unlike serial rsync processing, where you may only use a trickle of network because it’s checking each file one-by-one, rclone can fully utilize your bandwidth by working with many files in , whether that is disk-to-disk or disk-to-network.

There’s a nice blog post by Jeff Geerling that gives some examples.  He uses 32 threads for local disk-to-disk copies.  I’d never thought to use rclone for a local rsync before, but it’s what I’ll be reaching for in the future.