Horacio Reyes wrote: >>- The files are copied in batch of 100 tif images, in a directory with >>more than 5 million files. >> I have found thet SCP does not like lots of small files, if you can tar the files together into on big file it will probably transfer much quicker..
Do you know a command to tar through SSH which is the same as scp but kepps better permissions and is quicker?
Here it is:
You have bar/foo and you'd like to copy dir foo to server blabla: cd bar && tar c foo | ssh user@blabla "cd /dir_on_blabla && tar x"
Good Lord, that's complicated. A simpler suggestion:
# rsync -ave ssh /home/user/bar user@otherserver:/home/user/
If you don't have the rsync command available, you can install it from the rsync package with "up2date rsync". This command will take the bar directory (and all its subdirectories and files) and copy it over to otherserver using the "user" account. The "bar" directory will be placed *inside /home/user* (note the trailing slash). Rsync intelligently checks files to see whether they've been previously copied, and copies just the changed files so it is usually used to synchronize or mirror directories in different places; and of course, the command shown keeps all permissions and ownership intact.
Once you have the initial copy performed, you can put this command into a cron job every hour and keep your directories synchronized permanently. Rsync is also the basis for the excellent "rsnapshot" backup project on SourceForge.
Cheers,
-- Rodolfo J. Paiz rpaiz@xxxxxxxxxxxxxx http://www.simpaticus.com
