rsync with --fuzzy

• BackupAssist Support Forum • View topic - Rsync with --fuzzy
• http://www.snowfrog.net/2011/03/03/fuzzy-rsync/
• http://samba.2283325.n4.nabble.com/fuzzy-search-over-to-be-deleted-files-to-catch-moved-files-and-directories-td2511769.html – in this thread "henrik" AKA H. Langos-8 describes in a rather detailed way exactly what I was thinking of: renaming and moving files within the directory structure (actually my idea isn't restricted to "within …") w/o changing them (actually rsync-ing such files) should be made rather cheap; but that discussion dates back to 2009 …

My impression is, that --fuzzy writes the destination file again, making use of chunks from the old, neighbour destination file. That's certainly cheaper than sending files over the wire, but still … .

how to keep folders in sync over the Internet ("long-distance", not LAN)? I mean *sparingly*

So far I have been using rsync for this purpose, but there is still space for optimizations.

If I locally rename a big file, next time my "rsync -vaz --delete-after" will copy the file using the new name and remove it using its old name. I would actually prefer to see the file getting renamed remotely.

Posted this issue on the rsync user mailing list [link].
In response to that, I received a message pointing me to the "should" tool [link].

the IPKG optware development toolchain for Synology DiskStations

http://forum.synology.com/wiki/index.php/Overview_on_modifying_the_Synology_Server,_bootstrap,_ipkg_etc

Recently I found IPKG actually got abandoned, and the IPKG Perl on one of my Synology DiskStations wasn't really functional at all (some broken dependency on a library to be loaded dynamically).
That really frustrated me for a short period. Now I installed the IPKG optware-devel (compilation toolchain) and esp. gcc. Downloaded rsync sources, proceeded "as usually" with GNU sources, and it simply works. That's phantastic. I have really been worried for quite a while with that restricted Synology provided rsync, and the IPKG rsync didn't really comfort me there, so having "my own" rsync is really a nice thing. Next thing will be compiling my own Perl; and I will even be able to install CPAN modules with C sources, as I even have a C compiler.

Actually I should consider bootstrapping gcc, the GNU C compiler, because gcc-4.2.3 is a little outdated. gcc for my other DiskStation is even older, it's a gcc-3.x. But than those gcc-s create code, that runs on these DiskStations, and not really bad.

Looks like my Synology DiskStations even work well as development systems.

my new "mobile NAS": Synology DS112+

DS112+ Products - Synology Inc. Network Attached Storage

Installation procedure from CD-ROM:

• you are starting with a DiskStation with a bare disk, i.e. no DSM (the DS Management system)
• mount the supplied CD-ROM
• mount DSAssistant-SYNOLOGY_….dmg from the CD-ROM
• run the assistant App from there
• the assistant App helps you installing the DSM, either from a  DSM_DS112+_….pat file (around 40 MB large) on the CD-ROM (it comes with DSM-4.1) or from a file you download from the Synology website (bleeding edge: DSM-4.3)
• that includes formatting the hard disk

Installation procedure following following the quick set up leaflet:

• surf to find.synology.com
• it is meant to discover all reachable Synology DiskStations
• in this situation it does not find the DiskStation in question (running a DSM)
• it points you to DSAssistant-SYNOLOGY_….dmg, to be downloaded from the Synology website
• mount the downloaded DSAssistant-SYNOLOGY_….dmg
• run the assistant App from there
• …

Because I had temporary trouble accessing the supplied CD-ROM, I had to download the DSAssistant…, but I used the DSM-4.1 on the CD-ROM. That decision was unreasonable and done a little too fast, as I seriously intended to operate using DSM-4.3. So in the meantime I downloaded DSM-4.3, and after having set up DSM-4.1 entirely and properly, I set up DSM-4.3.

Further steps:

• setting up a personal user account
• allowing SSH access
• SSH log-in in using the root account
• modifying /etc/passwd to allow shell login for the user accounts in question (the resp. record has a "nologin" instead of a proper shell path)

Further steps:

• I followed the instructions listed in my own "my new NAS (…): Synology DS213+" (taken from somewhere else and my own experience)
• install the ipkg installation system
• install a proper rsync through ipkg; I will have to call this rsync as /opt/bin/rsync from now on
• "standard standard" installation is broken for the DS112+, but the wiki page describes the actual standard procedure including a fix in some bootstrap.sh
• commenting out the standard PATH setting in ~root/.profile and ~MY_ACCOUNT/.profile
• my first personal rsync-s to the DiskStation

my 1st concurrent backups from my Mac and my openSUSE Linux VM to the Synology DS213+

With the "alternative rsync" (see my older article on rsync and Synology!) things started looking a little more successful.

Apparently a lot of files by the name "@eaDir" get created "automagically" in various subdirectories. And as I usually start rsync with "--delete-after", they all get removed in the end. I am not sure, whether the DiskStation OS ("DSM") likes that, so for the time being I will do w/o that command line option. I will investigate the "@eaDir" issue "in the near future" (TBD).

Another issue: lots of messages like "cannot chgrp …". Apparently that's because the group identifier on the source system does not exist on the target system. I definitely don't want to see that message, so something must be done in that area (TBD).

It really is a relief to have a regular backup procedure in place again.

"rsync over ssh" to a Synology NAS – IPKG, opkg

Nota Bene / before:

• This is the text, that helps me personally re-installing rsync, whenever it magically disappears, maybe caused by an DSM update.
• True, I should bring a little structure into this article.
• I never ever care about rsyncd, as I am doing "rsync over ssh";  any activity suggested regarding rsyncd does not concern me. (But then sometimes I am misled, and I erroneously do invest time in false paths, and I do get into problems that are actually rsyncd-related.)
• I have my own ways of backing up data.
• I have my own ideas of keeping data long term outside my computer's built-in disks.

The Synology system provided rsync is not for me. It's somehow modified and spoilt.

backup - Rsync over ssh: "ERROR: module is read only" suddenly appeared - Server Fault

Well, it's not only me experiencing weird effects with the "built-in" rsync. And I think, quite a few of the answers to that forum article do not quite fit at all.

The details of the case:

• "rsync over ssh" does not use the remote rsync daemon at all, i.e. no rsyncd.conf applies
• it's not unlikely, that these rsync problems have to do with a Synology special version of rsync
• after this: "DiskStation> /usr/syno/etc.defaults/rc.d/S84rsyncd.sh stop", the error message does not change, so it's really not an "rsync daemon" issue, as that one is no longer running
• …

I fixed this problem by installing an rsync package provided on an Unslung Project repository for the CPU used within my Synology NAS. This brings dependencies on the Unslung Project, their IPKG installer, their Optware, their specific Synology repository. This kills me a little, to be honest.

I mainly followed the "German Synology wiki" on IPKG. (Anxiously as I am, I do keep a backup of that article at …/com/synology-wiki.de/ . Yes, I do have an idea, that this step alone is rather in vain.)
So, well 1st I had to install IPKG itself.
My CPU on the DS213+ is an e500 (actually: e500v2), so I went the e500 way in their description.
My CPU on the DS112+ is a "Feroceon 88FR131 rev 1 (v5l)", there is a note on how to deal with the DS112+, pls follow that one.

Then I installed the rsync package provided by IPKG.

An entry for /etc/rc.optware got created in /etc/rc.local, and they suggest to remove it, and I did that.
I am not sure, it concerns me at all, as I mention "my" rsync on the NAS by full path on my command lines outside the NAS.

Now my only minor problem with the "alternative rsync" is, that I have to specify the remote rsync on my local rsync command lines like this: "--rsync-path=/opt/bin/rsync". That's a PITA, but that's not too hard.

Update 2013-08-31: en.wikipedia.org/wiki/Ipkg says IPKG was discontinued. en.wikipedia.org/wiki/Opkg is the follower, code.google.com/p/opkg/, but it hasn't really taken off.

the wikis for the Synology DiskStation

• forum.synology.com/wiki
• www.synology-wiki.de – there is actually also a German wiki
• www.synology-wiki.de/index.php/IPKG – that's where and how I got hold on the alternative rsync

my new NAS (and so much more): Synology DS213+

DS213+ Products - Synology Inc. Network Attached Storage

The box makes itself known as diskstation.local resp. DiskStation on my LAN. (Reliability …)

Of course it deserves a "1 Gbit/s" on my FRITZ!Box.

The box's OS is BusyBox; here are a few more details:


DiskStation> uname -a
Linux DiskStation 2.6.32.12 #2668 SMP Tue Dec 11 13:01:16 CST 2012 ppc GNU/Linux synology_qoriq_213+


I enabled SSH access for my personal user account rather soon, but the SSH login was not instantly successful. I tried "ssh -v", and from there the login seriously looked, as if it was almost successful. My web research wasn't successful either. Rather desperately I enabled "telnet access"; I tried to log in, and this was the result:

login: can't run /sbin/nologin: No such file or directory

It was clear then, that the default shell in the /etc/passwd entry for my user account was in my way /sbin/nologin so far. I made /bin/ash the default shell and everything was fine. Of course, I disabled telnet access again thereafter. And of course, once I added "nologin" to my web query, I found a few articles describing the issue.

SCP works; "rsync over ssh" (with my personal user account) does not work yet:

$ rsync -va --rsh='ssh -l root' /etc/motd diskstation:tmp/ # works$ rsync -va /etc/motd diskstation:tmp/ # fails
sending incremental file list
ERROR: module is read only
rsync error: syntax or usage error (code 1) at main.c(1034) [Receiver=3.0.8]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.8]

Fiddling with /etc/ssh/sshd_config – I really don't know, what to change in order to overcome this obstacle.

Perl is not part of its standard equipment, but Python apparently is,

Through their Download Center I found this FTP server link. You can find various software for their boxes on their. There are also Perl installation packages, but the ones I tried are not for the current firmware release of "my" box, and in a way I am not up for an upgrade to beta software for my NAS.

(For now I summarised here everything I have tried so far, so I will be able to procede from here.)

A few more links:

• http://forum.synology.com/wiki
• http://forum.synology.com/wiki/index.php/What_kind_of_CPU_does_my_NAS_have
• http://www.synology-wiki.de/index.php/Welchen_Prozessortyp_besitzt_mein_System%3F (this article gives us an IPKG link for the DS213+)

There are suggestions on the wiki(s) to install ipkg (yet another packaging system, not officially supported by Synology), and to install another rsync through ipkg. I may consider this, but I am not yet sure. I am quite able to use rsync (over ssh) as user root, but I am not with my personal account. I have no clue, what rights it needs, to allow my personal account to work as smoothly as user root. (To Be Solved!)

To be continued …

QuickSSHd on my Android devices – pictures, e-books, … – created a few recipes for downloading and uploading

First: yes, QuickSSHd is very nice and useful software. Why does Android not come with something like that by itself?

Second: Yes, a graphical SFTP browser (such as Cyberduck) can be quite nice for interacting with an SSH server.

I have downloaded pictures from my tablet and my phone quite a few times now …
I have uploaded e-books to my tablet quite a few times as well ...

At some stage you just start thinking, doing it on the command line resp. in a script is your preferred way. I got there today, and I collected my one-liners in yet another tiny README / HOWTO document. That's a relief!

Actually I would like to rsync to my Android devices, but (I think) the rsync utilities available for Android don't act as remote party.

Update 2012-07-29:
The app's help screen describes briefly but exactly, how to create a new pair of ssh keys and how to transfer the public key to the other party. Basically the public key, from where you are going to log into you Android device, has to be listed in the ~/.ssh/authorized_keys on your Android device.

rsync, ionice, iotop

c't-Archiv, 23/2011, Seite 154: "Rsync zwingt Linux-Server in die Knie".

The above article (in German) is about using rsync together with ionice (like "nice" but for I/O load).
You can watch your processes' I/O load with iotop (like "top" but for I/O load).