(on Wikipedia)
https://rsync.samba.org/
Smart synchronization / excellent mirroring program
Though legendarily-difficult to comprehend, it does work well.
If you are a beginner, AVOID THIS PROGRAM! .. you will hate your life.
-
NOTE - I have included some very old notes.
- I've been using rsync with good success for backups. See my backup project.
-
Remote synchronization
--
- 2022-07-03 - Replaced by rclone
- 2020-03-25 - Some time ago I switched from using LFTP mirroring / reverse-mirroring to using rsync over SSH.
- 2016-03-28 - version 3.1.0 protocol version 31, on Slackware 14.1
- 2016-03-26 - 3.1.0 protocol version 31, on Lubuntu 14.04.4 LTS
- 2011-05-17 - (version not recorded), on (distribution not recorded)
-
rsync
comes with many distributions, and I've used it many times in the past.
- 1 2016-03-28 - version 3.1.0 protocol version 31
- 2 2016-03-26 - 3.1.0 protocol version 31
- 3 2011-05-17 - (version not recorded)
- 4 Rsync on Windows
- 5 Notes
- 6 Alternatives
2016-03-28 - version 3.1.0 protocol version 31 ∞
This came with Slackware 14.1
2016-03-26 - 3.1.0 protocol version 31 ∞
This came with Lubuntu.
Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, prealloc
2011-05-17 - (version not recorded) ∞
Basic rsync folder synchronization ∞
The first time I played with rsync it took me some time to figure out how to get it to work! To make the destination the same as the source, do:
\rsync -av --delete --progress /path/to/source/ /path/to/dest/
-a | --archive | Archive mode, equivalent to -rlptgoD - Yeah, that's a lot of options all rolled together. =/ |
-v | --verbose | Increase verbosity - Say more. |
--delete | Delete files that don't exist on the sending side - This one was very confusing. It means "Delete files on the destination which don't exist on the source" or I suppose another phrase would be "replicate deletions". |
|
--progress | Show progress | |
--bwlimit=KBPS | Bandwidth limit in kilobytes-per-second. - This is inexact, and most effective with large files. |
|
-n | --dry-run | Show what would have been transferred, but don't actually do anything. - Very important for testing! |
--exclude | Exclude/omit files. - You can use this parameter multiple times. |
(not working) Folder synchronization when the destination has limited free space ∞
Seems impossible.
nice -n 10 rsync --temp-dir=/path/to/rsync-temp/ --bwlimit=10000 --stats --sparse --delete-before --archive --hard-links --verbose --progress /mnt/source/ /mnt/dest/
Is --keep-dirlinks
useful?
Why the fuck isn't there just one --make-the-destination-the-fucking-same
parameter?
rsync folder synchronization over SSH ∞
See Rsync over SSH
rsync folder synchronization over FTP ∞
Rsync over FTP isn't possible. Why? I have no clue.. they decided to leave that part out..
Try lufs for mounting an FTP site, or LFTP and mirroring.
[not working] Merging directories ∞
NOTE! THIS DOES NOT SEEM TRUSTWORTHY!
It initially tested ok, but now it seems to destroy files!!
---
Finally, a decent replacement for mv!
rsync -av --progress --remove-source-files //source/* /dest//
I edit my .bashrc with this:
rsyncmv() { rsync -av --progress --remove-source-files "$1" "$2" ; }
so I can now do:
rsyncmv /path/to/source/* /path/to/dest/
---
possibly:
rsync -a SOURCE/ DEST/ --remove-sent-files --ignore-existing --whole-file
rsync troubleshooting ∞
permissions spam ∞
To remove them, instead of the usual -a
use -rlt
rsync cannot overwrite files ∞
sending incremental file list test.html rsync: rename "/mnt/mnt/httpdocs/.test.html.gpPO1R" -> "test.html": Operation not permitted (1)
This is because rsync incorrectly thinks that my local login would have to be the same uid/gid as the remote setup! WHOA.
No chmod voodoo is necessary. What you actually need to do is implement a workaround for rsync to pre-delete the files it would prefer to overwrite. Then the sync will be copying "new files" which will work just fine.
The solution is noted in secure synchronization. Gods that was annoying to implement.
This is probably a one-liner change in the rsync source for "don't be such a smartass".
rsync and cp cannot send large files over samba: File too large (27) ∞
See Samba
rsync: read errors mapping "/path/to/file.ext": Input/output error (5) ∞
rsync -av . /dest/
sending incremental file list path/to/file.ext rsync: read errors mapping "/path/to/file.ext": Input/output error (5) WARNING: path/to/file.ext failed verification -- update discarded (will try again). path/to/file.ext rsync: read errors mapping "/path/to/file.ext": Input/output error (5) ERROR: path/to/file.ext failed verification -- update discarded. sent 3578910 bytes received 2128 bytes 246968.14 bytes/sec total size is 79431643878 speedup is 22181.18 rsync error: some files could not be transferred (code 23) at main.c(1031) [sender=3.0.2]
Manually copying the file to a temporary location also gave an error:
cp: reading `/path/to/file.ext': Input/output error
But the file copied, and it was complete and proper.
So I copied the file from that temporary location to the destination and it gave no error!
Then rsync worked without error!
I have no idea what was going on here..
Rsync on Windows ∞
Ways to get it on Windows:
- cwRsync Free Edition for Windows.
-
Various Linux layers:
I read through stackoverflow.com/questions/528298/ but it's gone.. try https://stackoverflow.com/search?q=rsync+for+windows
"Operation not permitted" ∞
Various filesystems won't work properly. I've heard various things like to ignore these messages because things will work just fine, but I want rsync to work smoothly.
\rsync --recursive --progress --no-times --update --no-perms --no-owner --no-group --verbose /source/ /target/
As always, try the --dry-run
switch before running this for real.
To sync to a USB drive, I do:
\rsync --recursive --progress --no-times --update --no-perms --no-owner --no-group --verbose --delete --delete-before /source/ /target/
Notes ∞
Alternatives ∞
Last updated 2022-09-24 at 17:54:22
Added notes for rsync on Windows.
replaced by [[rclone]]