File Synchronization with Rsync over SSH

Nothing about rsync’s job has changed since 2009: it sends only the parts of files that differ, over SSH, with no daemon on the far side beyond sshd and the rsync binary. What has changed is small and worth knowing: SSH has been the default transport for years, so -e ssh is now noise; the -c checksum is no longer MD4; and --info=progress2 replaced the per-file progress wall. The commands below are the 2009 ones, cleaned up, and were run against a real sshd.

Push

rsync -avP ~/Music ~/Documents host2:~/

-a (archive) keeps permissions, times, symlinks and ownership where it can; -v lists what moves; -P shows progress and keeps partial transfers so a dropped connection resumes. The remote side is user@host:path; without user@ it uses your local user name.

sending incremental file list
Documents/
Documents/d.txt
Music/
Music/a.mp3

sent 267 bytes  received 65 bytes  664.00 bytes/sec

The trailing-slash rule from 2009 still decides everything: ~/Music (no slash) creates Music/ inside the destination, ~/Music/ copies its contents into the destination. Say it to yourself before pressing Enter.

Pull

Same command, sides swapped. Brace expansion still saves typing the host twice:

rsync -avP host2:~/{Music,Documents} .

Delete what the source deleted, after a dry run

--delete removes files on the destination that no longer exist on the source. Combine it with -n (--dry-run) first; the output is the exact list of what would be deleted:

rsync -avPn --delete ~/Music/ host2:~/Music/
rsync -avP --delete ~/Music/ host2:~/Music/

deleting a.mp3
b.mp3

Checksums instead of size and time

rsync’s quick check compares size and modification time. Edits that keep both, such as rewriting ID3 tags with --preserve-file-times, slip through; -c compares content instead. It reads every file on both sides, so use it when you suspect drift, not every night. rsync 3.2 and later negotiate the hash with the other side (xxHash where both ends have it, otherwise MD5); the 2009 text’s “128-bit MD4” is gone.

rsync -acvP ~/Music host2:~/

A single progress line, compression, another port

rsync -avz --info=progress2 ~/Music host2:~/
rsync -avP -e 'ssh -p 2222' ~/Music host2:~/

--info=progress2 prints one line for the whole transfer instead of one per file. -z compresses on the wire, worth it on slow links and pointless for MP3s or video. -e 'ssh -p 2222' is the modern reason to still spell out -e: options for ssh, such as a port or an identity file (-e 'ssh -i ~/.ssh/backup_ed25519').

Leave things out

rsync -avP --exclude '.cache/' --exclude '*.tmp' ~/ host2:~/backup/

Patterns are relative to the source directory; a trailing slash matches directories only.

Still one-way

rsync is unidirectional: whichever side you name as the source wins, and with --delete the other side loses. For two machines that both change, the companion recipe file-synchronization-with-unison-over-ssh covers the bidirectional tool.

To quickly synchronize files between two systems, rsync is an excellent tool that not only decreases the amount of time it takes to transfer files through a data deduplication algorithm, but can also be used transparently over SSH. The beauty of running rsync over SSH is that it does not require the rsyncd server to be running before a synchronization request and the connection is both authenticated and encrypted. All that is required is for the remote host you are connecting to be running the OpenSSH server component and of course the rsync application.

I use rsync the most for synchronizing my “Music” and “Documents” folders between a number of my systems at home and at work. All of these systems have these folders in the root of my home directory.

ls -ld ~/Music ~/Documents
drwxr-xr-x 16 gmendoza gmendoza 4096 2009-04-06 23:23 /home/gmendoza/Documents
drwxr-xr-x  9 gmendoza gmendoza 4096 2009-04-06 23:23 /home/gmendoza/Music

To push my recent changes from my local system (host1) to my remote system called (host2), I use the following commands.

rsync -avPe ssh ~/Music host2:~/
rsync -avPe ssh ~/Documents host2:~/

Notice, “Music” and “Documents” are specified without a trailing “/”, e.g. “Music/” or “Documents/”. This is important, because otherwise, it would copy only the contents of the folder to the remote home directory, and not the folder itself, which is described in more detail in the rsync man page.

Instead of running the above commands twice, you can also specify multiple files all in a single line.

rsync -avPe ssh ~/Music ~/Documents host2:~/

To synchronize changes made on the remote system to my local system, just reverse the commands. Notice the periods at the end of the line, which specifies the destination as the local working directory. Also, instead of wasting space by entering the host twice, you can use standard syntax to specify ranges or sets of files. In this case, I use curly brackets to specify the two directories on the remote host that share the same parent directory should be copied to my local working directory.

rsync -avPe ssh host2:~/{Music,Documents} .

I’ll also use the “delete” option to remove any files and folders the have been removed from the source system.

rsync --delete -avPe ssh ~/Music host2:~/

By default, rsync compares files extremely fast using a “quick check” algorithm based on the file size or in the last modified time (per the rsync man page). While I was updating my Music collection, I noticed that rsync was not detecting my ID3 tag modifications. By using the “-c” option, rsync will compare files using a 128 bit MD4 checksum as a more definitive change detection method. While this will slow the process down significantly, there’s obvious accuracy benefits in using the checksum method.

rsync -acvPe ssh Music host2:~/

Also, as you may have noticed rsync is strictly a unidirectional utility. This means that it only sends or receives data in a single direction, and it will clobber or delete any file or folder with the same name in the direction your are sending the data. For a great bidirectional utility, check out unison, which I will cover in an upcoming article.