rsync -a -z -v --partial --progress -e ssh <file_da_uplodare> <user_sourceforge>,<progetto>@frs.sourceforge.net:/home/frs/project/<prima lettera iniziale progetto>/<prime 2 iniziali progetto>/<nome completo progetto>/<directory dove uploadare>/

FRS Sourceforge.net

File copy over ssh is nice but it has no resume support. That can be a pain if dealing with big files and/or poor connectivity.

As a reminder to myself here is a shell alias I use :

alias scpresume='rsync -Pazhv -e ssh'

• -P is the same as –partial –progress, this enable progress indication and partially downloaded files transfer to be resumed
• -a activate archive mode to be recursive
• -z enable compression
• -h outputs human readable units
• -v add more verbosity, especially reassuring with poor connectivity
• ‘-e ssh’ is needed to use ssh as transport, you can change this to ‘-e “ssh -p XXXX”‘ if you need to change used port or any other ssh options

“”rsync tool kecil cabe rawit”"

Jika kata kunci yang anda ketikkan pada pakde google ‘backup tools enterprise’, maka yang muncul bacula.org dan amanda.zmanda.com keduanya adalah Tool Backup kelas Enterprise. Saya akan mencari tool yang keren untuk backup sistem CentOS, apapun dari mulai dari file konfigurasi, file svn, website, database Mysql pada intinya saya ingin tools yang ringan, mudah pemakaian, ringan di jaringan karena akan didistribusikan ke server/komputer lain via network, bisa sinkronisasi sewaktu terjadi kegagalan proses backup di tengah jalan, inilah tools yang saya cari. Mulailah percobaan saya lakukan dengan BACULA, instalasi cukup lancar walaupun dengan minimnya forum dan dokumentasi. Alhasil mulailah pada tahap konfigurasi, saya gagal karena harus dengan perintah/command untuk meng-konfigurasi, kemudian saya install webacula supaya lebih mudah tetapi jadi sulit seperti install bacula. Saya teringat telah menginstall webmin pada CentOS saya, apakah bisa saya konfigurasi dengan webmin? ternyata bisa dan mudah. Berlanjut dengan harapan agar cepat selesai dan segera on bacula. 1 hari lebih tidak juga dapat membackup dengan lancar, cukuplah saya harus cari tool yang lain, namanya juga Enterprise….

rsync jalan mudah, itulah mengapa saya pilih.

Cara menggunakan:
rsync -av –progress –exclude ‘*~’ –exclude ‘*.bak’ –exclude ‘.svn’ –exclude ‘*/tmp’ –rsh=’ssh -p2200′ root@192.168.0.100:/var/www/public_html/ /var/www/roojax.net/

Note: Error libacl.0.1 not share permison denied –> Selinux aktif
nano /etc/selinux/config ubah baris SELINUX=enforcing menjadi disabled

Rsync adalah tool yang berfungsi untuk transfer dan sinkronisasi file atau tree (struktur direktori dan file) secara satu arah, baik transfer lokal (di sistem yang sama) maupun remote (jaringan/internet). Fungsi rsync mirip/identik dengan tool-tool ini: cp, mv, scp, FTP client. Rsync biasanya digabungkan dengan SSH sebagai metode transpor remotenya, walaupun dapat juga disetup untuk menjadi daemon sehingga tidak membutuhkan SSH.  Dalam kasus-kasus tertentu rsync juga dapat digunakan menggantikan HTTP client seperti wget [campur.info]. Secara umum rsync sangat populer dalam membuat mirror server.

Pertama kali saya membaca tentang rsync di situs opensuse.or.id. Disana ada artikel mengenai “Backup Data dengan Rsync“. Penjelasan lebih detail tentang apa itu rsync, keunggulan-keunggulannya, perintah-perintahnya, silahkan kesini atau ke situs rsync ….

Saya menggunakan openSUSE 11.2, dan menginstall rsync dengan menggunakan tools zypper. Untuk instalasi di Slackware, silahkan lihat di sini, Ubuntu bisa lihat disini, distro lain silahkan googling
Ahmad-box:~ # zypper in rsyncSetelah proses instalasi selesai, selanjurnya melakukan konfigurasi
Ahmad-box:~ # vi /etc/rsyncd.conf

Khusus untuk konfigurasi di PC yang saya pakai, saya tidak mengikuti konfigurasi default, namun nyontek dari konfigurasi dari luar (mohon maaf….saya lagi-lagi lupa sumbernya). Begini isinya…

motd file = /etc/rsyncd.motd
max connections = 2syslog facility = local3
hosts allow = 10.0.0.0/24 10.0.2.0/24 12.1.4.0/24
# labelmodul:
[Ahmad]
path = /home/ahmad/Download/
comment = Ahmad punya
read only = yes
use chroot = yes
list = yes
uid = nobody
gid = nogroup
[iso]
path = /iso-dvd
comment = Iso Opensuse 11.2
read only = yes
use chroot = yes
list = yes
uid = nobody
gid = nogroup

Khusus untuk warna hijau dan orange adalah dua folder yang ingin saya bagi ke komputer lain.  Konfigurasi yang warna putih merupakan konfigurasi secara umum. Adapun file “/etc/rsyncd.motd” ini hanyalah berupa ucapan welcome untuk user…

motd file = /etc/rsyncd.motd
Welcome to Ahmad Rsync.....

File ini biasanya sudah ada, jika belum ada maka bisa dibuat sendiri. Selanjutnya restart service rsync
Ahmad-box:~ # service rsyncd restart

Testing dari komputer teman..

komputer@teman:~> rsync 10.0.2.107::
Welcome to Ahmad Rsync.....
Ahmad           Ahmad punya
iso             Iso Opensuse 11.2

Untuk copy data, berikut contoh perintahnya
komputer@teman:~> rsync 10.0.2.107::Ahmad/file /home/cl

(Sumber Artikel : http://alsinji.co.cc/2009/11/belajar-rsync/#awp::2009/11/belajar-rsync/ )

http://www.linuxforen.de/forums/showthread.php?t=204979

1. 在ubuntu上添加一个专门用于同步的用户组，root的权限太大，不建议对SSH开放。由于为了方便，rsync将使用不带密码的私钥进行登录验证，权限越低越好

groupadd remote-backup
useradd -c “account for rsync” -d /home/remote-backup/ -s /bin/sh -m -g remote-backup remote-backup

2. 利用cwrsync或putty产生一对密钥：

1) 使用cwrsync所带的ssh-keygen

sh-keygen -q -b 2048 -t rsa -f cwrsync -N “

将会产生不带密码的私钥cwrsync和公钥cwrsync.pub。把cwrsync.pub用加密的ftp上传到服务器上，放在/home/remote-backup下。

2)或者用puttygen，产生一对私钥和公钥，同样将公钥利用加密的ftp上传到服务器上，并执行ssh-keygen -if putty.key.pub > key.pub，将其转化为Openssh格式的。对于本地机器上的私钥，也要重新输出成为openssh格式的，并放到cwrsync的程序目录下。

3)编辑服务器上的公钥：

在ssh-dss前，加上这两句，如果本地电脑是动态IP，那就把from那句去掉

然后在/home/remote-backup下创建.ssh目录，把上面的公钥重命名为authorized_keys, 存放在.ssh下。

并确保.ssh是755, authorized_keys是644的属性。（假设其owner都是root, 如果是remote-backup自己的话，那权限就是700和600了）

4）编辑/home/remote-backup/validate-rsync.sh脚本，输入以下内容：

#!/bin/sh

case “$SSH_ORIGINAL_COMMAND” in
*\&*)
echo “Rejected”
;;
*\(*)
echo “Rejected”
;;
*\{*)
echo “Rejected”
;;
*\;*)
echo “Rejected”
;;
*\<*)
echo “Rejected”
;;
*\`*)
echo “Rejected”
;;
*\|*)
echo “Rejected”
;;
rsync\ –server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo “Rejected”
;;
esac

这个脚本可以确保以remote-backup身份登录的的用户，只能执行rsync这个命令，任何其他登录都会直接被切断。

chmod +x ./validate-rsync.sh

然后试运行一下，确保没有语法错误

3. 配置cwrsync:

将私钥放在程序目录下，和cwrsync.cmd在一起，，否则cwRsync会认为该私钥是共享的，需要600的权限，在Windows Xp下不好改。编辑cwrsync.cmd，在末尾加入

rsync -av -e “ssh -i cwrsync.key” remote-backup@mysite.com:/var/backup  /cygdrive/c/localbackup/

然后在命令行模式下运行cwrsync.cmd。第一次系统会提示，是否要将远程主机添加到信任列表中，选择yes。

然后就可以看到同步进行了。

成功后，可以在cwrsync.cmd中的v参数换成q，即rsync -aq ……。省去屏幕输出。

最后把cwrsync加入计划任务就可以定时同步了。

说明：通过ssh方式连接，就不在需要配置rsync daemon了。优点是所有传输内容都是保密的，缺点是尽管做了以上种种限制，resync还是可以看到全局的文档。一旦私钥被窃，别人是可以复制整个服务器上的文档。由于私钥是未加密码的，私钥的安全就很重要。所以本机的安全也就成了保护私钥安全的关键屏障。

只是考虑到，如果连rsync上的文档都被窃取了，服务器上的其他文档的重要性都还没有能跟他相比的，所以这个漏洞的意义不太大，也就不在进一步研究基于Openssh的chroot等方面的限制了。无论如何，采用私钥加密的方式，私钥和本机的安全是一定要注意的，建议要给Windows也设定一个非常安全的密码。

rsync配置成Daemon的话，还可以进一步参考这篇文章，里面介绍了rsync.conf的参数设定及其含义，DBA Note对公钥，私钥的认证过程也有简单的解释。多台window电脑间的同步也可以用rsync。

此外，更高级的使用方法，还有和inotify结合，当目标目录方式更新后，就主动同步。

本文主要参考文档：

Using Rsync and SSH

It’s just a quick tip. How can you force rsync to use a specific port when connecting to remote machine via ssh? Turns out it’s pretty easy, if you dig into man page deep enough. Simply specify needed port in ‘-e’ option like this:

rsync -av -e "ssh -p 123" example.com:/path/on/remote/server ~/local/path

As a bonus, with the same option you can add a lot more useful options, like compression and compression ratio:

... -e "ssh -p 123 -C -oCompressionLevel=9" ...

Da ich nun eine leere, verschlüsselte Festplatte habe und eine Backup-Festplatte die ebenfalls noch verschlüsselt werden will,

schreib ich hier mal ein kleines mini How-To, wie man sehr einfach die Daten der einen Festplatte auf die Andere bekommt.

Sehr einfach geht das z.B. mit “rsync“

$ rsync -avogtplrz  /media/disk-1/ /media/disk/

“disk-1″ ist hier meine Backup-Festplatte, mit den Daten die Kopiert werden wollen. “disk” ist die leere Festplatte, die nun mit den Daten von “disk-1″ gefüllt wird.

A nice little tool I came across – DeltaCopy – for when you have to work with Windows desktops and want to rsync (and don’t want to install Cygwin). I haven’t had time to play with it yet, but apparently the DeltaCopy client will connect to an rsync server.

Out of interest, there’s also a SourceForge package for installing a minimal ssh server on Windows, rather than the full Cygwin.

Stamattina ho scritto come sia possibile usare le regole di Udev per far si che i dispositivi vengano montati in una directory piuttosto che in un’altra a seconda del dispositivo. Ora vedremo come aggiungere regole a Udev per fargli eseguire un determinato script quando si monta una penna USB (o quando la si smonta).

Prima di tutto dobbiamo ottenere informazioni sull’hardware, quindi diamo questo comando da terminale:
# tail -f /var/log/syslog
e inseriamo la nostra periferica USB (penna, HD esterno). Otterremo un bel po’ di testo come output, segniamoci il percorso della device (ad esempio /dev/sdb).

Ora che sappiamo dove trovare la periferica, possiamo ottenere molte informazioni, facciamolo con questo comando:
$ udevinfo -a -p $(udevinfo -q path -n /dev/sdb)

Anche in questo caso come output otterremo molte righe di testo, ma tra questi, i dati che ci servono non sono molti, segnateveli, perche’ vi serviranno per scrivere la/e regola/e.

Un file di regole per Udev e’ ad esempio questo:

---8<---
BUS=="scsi", SYSFS{model}=="BLAblabla", KERNEL=="sd?1", NAME="pendatilavoro", OWNER="luca"
SUBSYSTEM=="block", ACTION=="add", RUN+="/etc/udev/dev.d/script/mount.dev"
SUBSYSTEM=="block", ACTION=="remove", RUN+="/etc/udev/dev.d/script/umount.dev"
---8<---

da salvare ad esempio su /etc/udev/rules/10-dam.rules (il numero davanti al nome indica la priorita’ di esecuzione, piu’ basso e’ il numero prima verra’ eseguita la regola).

Questo e’ il significato dei valori:

BUS=="scsi" =>                      la penna USB viene riconosciuta come disco scsi
SYSFS{model}=="UMni512MB*IOM2J4" => un modo per riferirsi a quella specifica penna USB
KERNEL=="sd?1" =>                   la prima partizione della penna
NAME="pendatilavoro" =>                  Udev mettera' la nostra penna nel percorso /dev/pendatilavoro (per la prima partizione)
OWNER="luca" =>                    il percorso /dev/pendatilavoro e' dell'utente luca

SUBSYSTEM=="block", ACTION=="add", RUN+="/etc/udev/dev.d/script/mount.dev" => quando la periferica viene inserita viene eseguito lo script RUN+="/etc/udev/dev.d/script/mount.dev"

SUBSYSTEM=="block", ACTION=="remove", RUN+="/etc/udev/dev.d/script/umount.dev" => quando la penna viene estratta viene eseguito lo script RUN+="/etc/udev/dev.d/script/umount.dev"

Aggiungendo script, potete ad esempio fare in modo che la penna si sincronizzi con una directory del vostro hard disk (potete utilizzare rsync) o quando la inserite, o prima di smontarla. Con rsync il comando e’ una cosa del genere:
$ rsync -a -v --delete /dev/pendatilavoro /home/utente/hddatilavoro
quindi con le dovute modifiche fate lo script e inserite il suo percorso nella regola di Udev.

Fonte: Udev – action add – remove

[^] torna su | post<li> | 

A BSRSoft acabou de disponibilizar a seus clientes e ao mercado em geral o seu novo serviço de backups para servidores de qualquer tipo, backup de dados empresariais e dados pessoais.

O sistema utilizado neste novo serviço é igual ao sistema que utilizamos para efetuar backup de todas as nossas operações críticas (hospedagem web de clientes, backup de servidores de banco de dados de nossos clusters etc). É o NetBAQ.

Alta confiabilidade, alto desempenho, altíssimo potencial de escalabilidade, baixo custo (você se livra da responsabilidade de guarda de backups e do investimento em infraestrutura para guarda segura de seus dados) e muito mais proporcionado pelo conceito de prestação de serviços hosteados na nuvem (cloud computing).

O sistema pode ser acessado via VPN, rsync (sob SSH criptografado), FTP, mapeamento transparente de drive via rede (via VPN) e através do SAMBA.

Para acessar as opções e verificar os valores, visite: http://bsrsoft.com.br e clique em “BackUP” no menu superior, ou então entre em:

http://shop.bsrsoft.com.br/category.php?id_category=17

Precisei sincronizar algumas pastas do nosso servidor principal para um servidor secundário. Utilizamos dois servidores Ubuntu 8.04. A solução veio com o Rsync, que já vem instalado por padrão nesta distribuição.

No computador destino criei a pasta /home/tiago/sincronizacao-servidor.

Lendo o man do rsync cheguei a conclusão que, para o meu caso, alguns parâmetros serão úteis:

v - verbose
-r, --recursive recurse into directories
-u, --update skip files that are newer on the receiver
-h, --human-readable output numbers in a human-readable format
-i, --itemize-changes output a change-summary for all updates
-P same as --partial --progress
-t Preserva a data original do arquivo
--delete-excluded


Então minha linha de comando ficaria mais ou menos assim:
rsync -vtruhiP --delete-excluded /var/www/ tiago@192.168.0.3:/home/tiago/sincronizacao-servidor/


Este comando funciona corretamente, mas pede a senha do usuário tiago na máquina destino. Como desejo que o rsync seja executado sem minha intervenção, precisarei de uma solução para isto.

Na máquina origem execute a seguinte instrução (no meu caso como usuário root):
ssh-keygen -t dsa -f ~/.ssh/id_dsa
E vá teclando enter nas opções que for pedido. Não informe nada.

Copie o arquivo criado /home/tiago/.ssh/id_dsa.pub para o servidor destino
scp /root/.ssh/id_dsa.pub tiago@192.168.0.3:/home/tiago/.ssh/

No servidor destino, execute o seguinte comando
cat /home/tiago/.ssh/id_dsa.pub >> /home/tiago/.ssh/authorized_keys

Execute o comando rsync novamente e veja que não é mais solicitado senha. Se ainda pedir, reinicie os servidores. Caso reinicie e continue pedindo, execute todos os passos anteriores deste post novamente.

Criei um script para executar o rsync e criar um log disso (script achei na internet, citarei as fontes no final)

vim executa_sincronizacao.sh
E inseri neste arquivo o seguinte código:

#!/bin/bash
INICIO=`date +%d/%m/%Y-%H:%M:%S`
LOG=/var/log/rsync/`date +%Y-%m-%d`.txt
echo " " >> $LOG
echo " " >> $LOG
echo "|--------------------------------------------" >> $LOG
echo " Sincronizacao iniciada em $INICIO" >> $LOG
rsync -vruhiP --delete-excluded /var/www/ tiago@192.168.0.3:/home/tiago/sincronizacao-servidor/ >> $LOG
FINAL=`date +%d/%m/%Y-%H:%M:%S`
echo " Sincronizacao Finalizada em $FINAL" >> $LOG
echo "|--------------------------------------------" >> $LOG
echo " " >> $LOG
echo " " >> $LOG

E dei permissão para execução no arquivo>
chmod 775 executa_sincronizacao.sh
Criei a pasta rsync dentro de /var/log/, para sejam criados os logs nesta pasta:
mkdir /var/log/rsync

Depois disso coloquei no crontab para executar o executa_sincronizacao.sh de tempos em tempos.

Espero que este post seja útil:

Fontes consultadas:
http://www.dicas-l.com.br/dicas-l/20041227.php
http://www.guiadohardware.net/comunidade/sincronizar-pastas/942084/

No soy un usuario muy asiduo de rsync; sin embargo las características de esta aplicación son muy completas así que vale la pena probarlo. Hace algunas semanas encontré la versión de rsync para Windows y me decidí a usarla para sincronizar los documentos entre mi laptop y una memoria USB.

¿…y entonces por qué este post? porque en Windows es bien sabido que las unidades de disco se nombran con una letra seguido de 2 puntos (:), pero en rsync, como proviene del mundo *nix, se utilizan para separar el usuario y el host, del directorio físico (al estilo user@host:/directorio).

¿La solución? utilizar la siguiente notación: supongamos que queremos sincronizar 2 directorios ubicados en el disco C: y en el disco D:

rsync --update --recursive --progress /cygdrive/c/directorio_origen/* /cygdrive/d/directorio_destino/

intipadi.com – Zsync adalah sebuah program file transfer yang hampir sama dengan rsync. Zsync sendiri sebenarnya digunakan untuk mengoptimasi suatu distribusi file via internet dengan satu atau lebih server kemudian dengan massive dapat disebarkan ke ribuan downloader yang butuh.

untuk menginstall Zsync di Ubuntu, silakan gunakan perintah:

sudo apt-get install zsync

kemudian pada artikel ini akan kita coba mengupdate ISO Karmic Koala kita supaya sesuai dengan update terbaru. Caranya cukup jalankan Zsync dengan perintah:

zsync -i karmic-desktop-i386.iso http://releases.ubuntu.com/karmic/ubuntu-9.10-beta-desktop-i386.iso.zsync

Sekarang ISO Karmic anda sudah terupdate, sama dengan versi terbarunya di server. Selamat mencoba

I became desperate yesterday night to do some useful things, anything. Had a long offline Puja Holidays. I was struggling to understand “How to write Makefiles”, and yesterday it clicked and I ended up writing a Makefile for the ongoing fedX project. Though the makefile which I wrote is not that great, but I have a start now. Committed the changes to the fedX repository at

http://gitorious.org/~rtnpro/fedx/rtnpros-fedx-clone/

It was around 4:00 AM in the morning already. Made some changes in my gitorious project clones. Apart from that, I was trying to download the rpmfusion repositories for Fedora 11. I was using rsync on multiple screens for the purpose. But again and again, rsync used to stop downloading after working for sometime. Then I tried a Firefox addon “DownThemAll” and was very happy to see it work. All I needed was to browse the repository page listing all the packages, right click and select DownThemAll. And then I just needed to select where I want to save the downloaded files and start it. It downloads multiple files simultaneously and does not halt when one of the files cannot be downloaded, rather it pauses it and moves to others. It also supports resume, and has an option to skip the already downloaded files.

But I could not download the directories with DownThemAll. I guess it works only for files. Need to do some research on it. Any way, you can try to use DownThemAll. I think you will like it.

Finally, I went to sleep at 4:45 AM with my laptop running DownThemAll to download the rpmfusion repository. When I got up today morning, it was running fine.

On the server to be backed up:

1. Install rsync
2. Create /etc/rsyncd.conf

The simplest rsyncd.conf file:

[backup]
path = /var/www/www.redhat.com/

Next run rsync in daemon mode: ]$ rsync –daemon

On the server pulling the backup data:

In the contrab (could be done manually):

/usr/bin/rsync -a --delete --include=*.html --include=*.png --exclude=*.* server.name.here.com::backup /where/to/save/

Backup on Ubuntu.

Updated 09/11/09 to correct some errors!

This “How To” was written with the help of this article by Troy Johnson, here and amended were necessary to fit my needs.

I have 2 PC’s on a network and want to backup some files from one PC the [remote_client] to the other PC the [server] to do this I want to run a script from the server. The process can be atcheved using rsync & SSH and automated with a cron job on the server.

Step:1

We install on the server

open-ssh-server and rsync

and on the client openssh-client and rsync

This can be done via the Synaptic Package Manager or using the command line i.e.

~$ sudo apt-get install openssh-client

~$ sudo apt-get install rsync

you also may want to check that cron is installed and if you want a GTK version of rsync you could install gadmin-rsync.

So the files I want to backup are on the remote_client in the following folder; /home/martin/my_files

and I want to back them up on the server in the following folder; /home/martin/backups

Note: I have an account on both PC’s with the same name and password, and read access to both the source & destination folders.

On the server we test out a basic script called backup.sh and saved with the Permissions set to “allow executing as a program”, check out my post here for help with that, the content of the script is;

#!/bin/sh

## first script to copy files from remote_client to server ##

rsync -avz -e ssh martin@remote_client:/home/martin/my_files/ /home/martin/backups

save the script and then run it by typing;

~$ ./backup.sh

in a Terminal assuming you saved the script in your home directory /home/martin and called it backup.sh. You should be prompted for your user password on the remote_client then the process should start.

To make this more secure edit the ‘authorized_keys’ file in servers /home/martin/.ssh folder with a text editor and modify the line with ’server-rsync-key.pub’ information on it from this:

Step:2

Ok so that does the trick but if I’m to automate this script it needs to work without SSH asking me for my password each time. To do this I need to generate a private/public key pair, this is much more secure than adding the actual password in the text of the script or elsewhere linked to it.

We now need to be logged onto the server;

To generate the key pair I open a [Terminal] and issue the command;

~$ mkdir /home/martin/.key

to create a hidden folder in my home folder called .key [the dot makes it hidden], then;

~$ ssh-keygen -t dsa -b 1024 -f /home/martin/.key/server-rsync-key

Generating public/private dsa key pair.

Enter passphrase (empty for no passphrase): [press Enter]

Enter same passphrase again: [press Enter]

Your identification has been saved in /home/martin/.key/server-rsync-key.

Your public key has been saved in /home/martin/.key/server-rsync-key.pub.

The key fingerprint is:

etc. etc.

I now have 2 files in the .key folder in my home folder called;

server-rsync-key

and

server-rsync-key.pub

I now copy the server-rsync-key.pub key over to the remote_client into my home folder /home/martin and log onto the remote_client.

Now on the remote_client, I issue the following command in a [Terminal]

~$ if [ ! -d .ssh ]; then mkdir .ssh ; chmod 700 .ssh ; fi

[this checks to see if the folder .ssh exists and if not creates it]

~$ mv server-rsync-key.pub .ssh/

[this moves the key from my home folder to the .ssh folder, hidden in my home folder]

~$ cd .ssh/

[we now move to the .ssh folder]

~$ if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi

[this checks to see if there is an authorized_keys file and if not creates it with the correct permissions]

~$ cat server-rsync-key.pub >> authorized_keys

[this copies the contents of server-rsync-key.pub key into the file authorized_keys file.

If we now run our original script with the amendment below on the server it no longer asks for a password.

rsync -avz -e "ssh -i /home/martin/.key/server-rsync-key" martin@remote_client:/home/martin/my_files/ /home/martin/backups

Step:3

Now the key can be used to make connections to the remote_client, but these connections can be from anywhere (that the ssh daemon on remote_client allows connections from) and they can do anything (that remote_user can do), which could be dangerous.  To make this more secure edit the 'authorized_keys' file in remote_client /home/martin/.ssh folder with a text editor and modify the line ending user_name@server_name from this:

ssh-dss
yl6b2/cMmBVWO39lWAjcsKK/zEdJbrOdt/sKsxIK1/ZIvtl92DLlMhci5c4tBjCODey4yjLhApjWgvX9
D5OPp89qhah4zu509uNX7uH58Zw/+m6ZOLHN28mV5KLUl7FTL2KZ583KrcWkUA0Id4ptUa9CAkcqn/gW
kHMptgVwaZKlqZ+QtEa0V2IwUDWS097p3SlLvozw46+ucWxwTJttCHLzUmNN7w1cIv0w/OHh5IGh+wWj
V9pbO0VT3/r2jxkzqksKOYAb5CYzSNRyEwp+NIKrY+aJz7myu4Unn9de4cYsuXoAB6FQ5I8AAAEBAJSm
DndXJCm7G66qdu3ElsLT0Jlz/es9F27r+xrg5pZ5GjfBCRvHNo2DF4YW9MKdUQiv+ILMY8OISduTeu32
nyA7 etc....

to this:

from="10.1.1.1", command="/home/martin/.key/validate-rsync" ssh-dss bA402VuCsOLg
yl6b2/cMmBVWO39lWAjcsKK/zEdJbrOdt/sKsxIK1/ZIvtl92DLlMhci5c4tBjCODey4yjLhApjWgvX9
D5OPp89qhah4zu509uNX7uH58Zw/+m6ZOLHN28mV5KLUl7FTL2KZ583KrcWkUA0Id4ptUa9CAkcqn/gW
kHMptgVwaZKlqZ+QtEa0V2IwUDWS097p3SlLvozw46+ucWxwTJttCHLzUmNN7w1cIv0w/OHh5IGh+wWj
V9pbO0VT3/r2jxkzqksKOYAb5CYzSNRyEwp+NIKrY+aJz7myu4Unn9de4cYsuXoAB6FQ5I8AAAEBAJSm
DndXJCm7G66qdu3ElsLT0Jlz/es9F27r+xrg5pZ5GjfBCRvHNo2DF4YW9MKdUQiv+ILMY8OISduTeu32
nyA7 etc....

where "10.1.1.1" is the IP address of server, and "/home/martin/.key/validate-rsync" point to a script called validate-rsync as follows:

#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
*\|*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac

If server has a variable address, or shares its address (via NAT etc.) with hosts you do not trust, omit the 'from="10.1.1.1",' part of the line (including the comma), but leave the 'command' portion. This way, only the 'rsync' will be possible from connections using this key. Make certain that the 'validate-rsync' script is executable by remote_user on remote_client and test it.

So in my case the result looks like this:

command="/home/martin/.key/validate-rsync" ssh-dss bA402VuCsOLg9YS0NKxugT+o4UuIj
yl6b2/cMmBVWO39lWAjcsKK/zEdJbrOdt/sKsxIK1/ZIvtl92DLlMhci5c4tBjCODey4yjLhApjWgvX9
D5OPp89qhah4zu509uNX7uH58Zw/+m6ZOLHN28mV5KLUl7FTL2KZ583KrcWkUA0Id4ptUa9CAkcqn/gW
kHMptgVwaZKlqZ+QtEa0V2IwUDWS097p3SlLvozw46+ucWxwTJttCHLzUmNN7w1cIv0w/OHh5IGh+wWj
V9pbO0VT3/r2jxkzqksKOYAb5CYzSNRyEwp+NIKrY+aJz7myu4Unn9de4cYsuXoAB6FQ5I8AAAEBAJSm
DndXJCm7G66qdu3ElsLT0Jlz/es9F27r+xrg5pZ5GjfBCRvHNo2DF4YW9MKdUQiv+ILMY8OISduTeu32
nyA7 etc....

Our backup.sh script now looks like this:

#!/bin/sh

## second script to copy files from remote_client to server ##

rsync -avz -e "ssh -i /home/martin/.key/server-rsync-key" martin@remote_client:/home/martin/my_files/ /home/martin/backups

Step:4

I now need to run the script from a Cron job, first move the backup.sh script into your $PATH, for help with this check out my post here.

Now on the server open a [Terminal] and issue the command:

~$ crontab -e

This should open a text editor, normally vi, normally paste into it the following text as a reminder

# leave a space between each option #
# the * is a wildcard option #
# dayofmonth is 1 to 31 #
# dayofweek can be 0 – 6, 0 is Sunday #
# or use mon tue wed etc #
#
# mins hours dayofmonth month dayofweek command
# * * * * * * command

Then use Crontrol+o to save and press [Enter] to save, then Control+x to [Exit].

and test…

I was particularly impressed by Dropbox and its ability to backup, retrieve and share files so easily, but I am quite concerned about the security issues deriving from having your files on a remote, closed-source system. I think that a good remote storage solution should prevent the owner of the storage server from accessing private data. Is it even possible? It is, using the right approach: the client computer must perform encryption when storing remote files, and decryption when retrieving them. I put together a solution for Linux systems using sshfs and encfs.

I used two computers: a server and a client, but for testing purposes just one computer is needed. The server acts as a storage provider, using an ssh server. It can serve multiple users, and each user has a box where he can put anything. A user that connects to the server creates an encrypted space using encfs, and fills the box with private data.

On the server

• Add a user, for example remote_user1
• Prepare a directory that will be the remote_user1 personal box. For example:

# mkdir /srv/boxes
# mkdir /srv/boxes/remote_user1
# chown remote_user1.remote_user1 /srv/boxes/remote_user1

• Install ssh server. On Debian-based systems:

# apt-get install open-ssh-server

• edit /etc/ssh/sshd_config to include the following options (this will prevent the users to use ssh to do anything but file transfer on their personal box):

Subsystem sftp internal-sftp
ChrootDirectory /srv/boxes
ForceCommand internal-sftp

• Reduce read/write permissions on the root server directory. This is done to prevent the user to mount the root directory and list its contents.

# chmod 711 /srv/boxes

• Restart ssh server, for example:

# invoke-rc.d ssh restart

On the client

• You can add a user, for example user1, or continue with an existing user.
• Install ssh client, sshfs and encfs. On Debian-based systems:

# apt-get install open-ssh-client sshfs encfs

• Add user1 to the fuse group

# adduser user1 fuse

• Log in as user1 (if you are already logged in as user1, log off and log in to render the fuse group effective)
• Create a folder that will contain the encrypted file system mounted through sshfs

$ mkdir ~/.box_enc

• Mount the remote directory, providing remote_user1 password (boxserver is the computer where we created the remote boxes before)

$ sshfs remote_user1@boxserver:remote_user1 ~/.box_enc -o uid=$(id -u) -o gid=$(id -g)

• Create a folder that will contain the unencrypted file system mounted through encfs

$ mkdir ~/box

• Mount the encrypted directory, creating an encfs environment encrypted with a password. After this step, the directory ~/box can be used as any other directory, but everything in it will be encrypted and stored in a remote folder, specifically the /srv/box/remote_user1 folder in your server. If you list the content of that remote folder (that is also the content of ~/.box_enc folder in the client computer) you will only see encrypted content and the xml file that encfs uses to keep the environment settings.

$ encfs ~/.box_enc ~/box

• To unmount the directories, execute, in order:

$ fusermount -u ~/box
$ fusermount -u ~/.box_enc

Tips

• You can setup ssh with key authentication instead of password authentication to avoid typing the password every time you want to mount the encrypted remote folder. To do so, on the client side:

$ ssh-keygen
$ cp ~/.ssh/id_rsa.pub ~/.box_enc #~/.box_enc must be mounted

On the server side, logged in as remote_user1:

$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ cat /srv/boxes/remote_user1/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys
$ rm /srv/boxes/remote_user1/id_rsa.pub

• You can setup encfs to read the password from a file, using for example:

$ read -s -p 'password>' pwd
$ touch ~/.encfs_pwd
$ chmod 600 ~/.encfs_pwd
$ echo "$pwd" >> ~/.encfs_pwd
$ pwd=
$ encfs --extpass='cat ~/.encfs_pwd' ~/.box_enc ~/box


• You can use the mounted directory as a backup, and use rsync to synchronize a local folder and your backup folder. For example:

$ #To pull your files from your backup folder:
$ rsync -uav ~/box/ ~/Documents/
$ #To push your files to your backup folder:
$ rsync -uav ~/Documents/ ~/box/

• Then you can schedule a job to keep them synchronized. To run a sync every minute:

$ crontab -e
* * * * * rsync -ua /home/user1/box/ /home/user1/Documents/ && rsync -ua /home/user1/Documents/ /home/user1/box/

• To change the behavior of the server so that only a certain group of users are restricted to the file transfer functionality of ssh, you can use the Match keyword inside the sshd_config file, before the “ChrootDirectory” and “ForceCommand” lines. (see “man sshd_config”)

Security

The security implications of this setup are complicated. Mostly, this setup is as secure as the combination of encfs and your client computer. The remote server cannot see the content of your files unless it knows the encfs password, but it can see the number of files and the directory structure (see http://www.arg0.net/encfsintro and encfs man pages for more details). Theoretically, it could be possible for someone with access to the remote server to understand what kind of files you have on your box. For example, a big 400MB file could be a movie, a compressed file or a CD image; a directory could have a structure that resembles that of a Firefox profile folder.

Usability

The setup I just described is quite easy to follow if you know Linux well. It is tricky if you’re not expert. This solution can be automated in some aspects, for example an installation could setup the remote box to mount at login. Moreover, a nice GUI could be developed to do a quick start, to select backup options and security settings. The performance is not so great, since the data must be encrypted two times: one for encfs and one for ssh; it is however faster than I thought (I have not done benchmarks); moreover both sshfs and encfs have options (like cache and compression) that could speed up the file transfer. About the Operating System: this setup works on Linux, but I think it should work also on Mac OS X thanks to the macfuse project that enables, among others, sshfs and encfs. On Windows sshfs is available through Dokan libraries, but I can’t find anything for an encfs port.

Final considerations

This experiment is satisfying from a technical point of view, but it lacks the usability to be considered a full-blown alternative to Dropbox; I think that a better alternative could be Novell’s iFolder. Currently, from what I found, iFolder server kinda works on non-Novell distros, but it needs some hacks and the installation is not straightforward (see http://ubuntuforums.org/showthread.php?t=1163192&page=3), but it’s promising. I think I will try to install a complete setup when I have the time.

[Update]

I found a blog that reports stability problems with sshfs + encfs setups. I haven’t found the timestamp problem he mentions, but the two computers I used are tightly connected. I didn’t do a “heavy load” stress test either, just syncronizing a directory containing the sources and binaries of a small project.

:: :: :: :: :: :: :: :: :: :: :: :: :: :: :: ::