Auto-rsync local changes to a remote server

Submitted by fago on Fri, 04/11/2014 - 18:05
As I've never been happy with the performance of Drupal running on behalf nfs shares in vagrant, I've been looking into a simple way to rsync my local changes into a vagrant box. There are separate tools like lsyncd or the deployment features of IDEs, but I wanted to have a simple solution that is easily scriptable. Thus, I ended up writing a simple shell script that watches a directory for changes events using inotify-tools and rsyncs the changes to a remote system via SSH.

watch.sh

The simple combination of inotify-tools and rsync ends up being problematic with changes that happen during a sync operation. There are simple approaches like the script discussed over there, but that involves a delay and still has the possiblity of missing changes. Therefore, I figured there need to be at least two processes when the sync operation is active: One process consumes file change notifications and processes them, while another process handles the synching. If change events come in during processing, the events are aggregated - i.e. only one event is propaged to the synching process, which then repeats the sync once the initial sync is finished. Of course, changes coming in during the repeated sync operation are again aggregated - so the script keeps syncing changes to the remote system until the chain of file system notification ends. This file event processing logic is all handled by a small script called watch.sh, which can execute any command based on the file system changes. As mentioned, the script makes use of two processes for handling event notifications and running the command, while it makes use of named pipes for letting the processes communicate. You can simply run watch.sh with rsync as follows
./watch.sh rsync -Cra --out-format='[%t]--%n' --delete SOURCE TARGET
Turns out, this simple script works quite reliable. However, when using it repeatedly with different commands suiting for multiple projects, a neat way for firing and logging it would be useful. Because of that, I wrote a simple initd like wrapper script that starts/stops the watch.sh script.

syncd

Syncd is a simple bash script that ships with the watch.sh script and makes configuring it for auto-rsyncing local changes simple. It comes with an initd script called syncd that reads syncd.conf configuration files, whereas it looks for syncd.conf files in the current working directory or any parent directory. That way, this works similar to vagrant & Vagrantfiles - just put syncd.conf somewhere in a parent directory "syncd" will pick it up. Known commands are syncd run to manually trigger a rsync and syncd log to inspect recent rsync output, besides the regular initd arguments start, stop, restart and status. I've put the syncd script (bundling watch.sh) on github, please feel free to use it and share your thoughts. The script is linux-only right now as it makes use of inotify-tools, but it should be rather easy to support Max OS X via conditionally making use of fsevents-tools instead (PRs welcome!). Update: If you are interested in using syncd to auto-rsync changes to vagrant boxes as well, please see the respective blog post.

Tags

Wim Leers (not verified)

Sat, 04/12/2014 - 11:18

Nice work, this looks like a very handy tool :) I especially like it because it is so simple. Having written http://fileconveyor.org/, I know how complex it can become. You chose a very useful subset of functionality, and relied on the shoulders of known robust tools: inotifywait and rsync. File Conveyor does much more, but is therefore also much more complex in setup: it's cross-platform, written in Python, is intended to run as a daemon, can write to anything that you want (e.g. S3), can perform processing on the file before sending it to its destination and rembers the URLs of the written files (it was written for CDN integration). That's why I like this so much: very narrow use case, but therefore infinitely simpler and probably more reliable.

Yeah, it's a simple script that fits a specific use-case. I figured a short script might be easier to handle than more complex solutions that need to be configured and revised until they match the requirements - e.g. I could easily handle aggregated events as suiting for that use-case.
I've considered checking out File Conveyor also, but as you say - it does more and is built for a different use-case. It seems to be a very handy tool for what it's built though :-)

Gueno (not verified)

Thu, 12/11/2014 - 11:36

Hello and thanks a lot for this simple but strong tool. (I'm french so excuse me for my poor english) : Is there a way to sync a dir to multiple servers ? In my case, I need to copy files on 2 servers. Another suggestion : why did you use --delete param by default instead of use it in params var of your script ? By default, this param is not enabled in rsync and I think its dangerous to let it activated. Thanks for your response :)