Re: Best practice for bulk data transfer

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Friday 17 Feb 2006 08:00, Andrew wrote:
> I just need to transfer a couple hundred thousand files off a **REALLY**
> slow server to a faster server and to be able to document that file x on
> slow server is identical to file x on faster server. It is not a forensics
> problem per se -- but I do need to prove/document the process -- so I'm
> thinking md5sums.
>
> /** begin simplification **/
>
> Procedure 1: The slow option
>
> [slow server] calc md5sums > list && nc list > faster_server
> [slow server] rsync files to faster@server over a 10T crossover cable
> [faster server] check md5sums against list
>

This means you have to boot the OS on the slower machine.  NO.  The suspicious 
minded will say "What if the person who controlled that machine left a 
software bomb thats ready to go off if the machine is booted?"

Script pseudocode
1] Start program when machine boots
2] wait 10 minutes
3] if someone with the right permissions doesn't notice me and kill me first, 
delete all those suspicious files.

I have something similar on a laptop designed to be used by a Partner at a law 
firm.  Script runs even in level 1.
If anyone other than her logs in, OR she doesn't log in within 10 minutes 
after the system boots, her entire home directory gets encrypted with gpg.
If she forgets, tough .. I have her gpg keyring backed up :p, she has to come 
to me to get her data back.  
Not perfect, I know, since I could strip the entire home partition in 30 secs 
(as I'm sure most of the posters here could do), but it does stop the data 
being at risk from the casual thief who is more likely to nick the laptop, 
turn it on and go browsing around.


> Procedure 2: The faster option (repeated for each hard drive)
>
> [slow server] halt
> pull hard drives off slow_server and place in removable tray on
> faster_server
> [faster server] dd if= /dev/slowdrive ibs=4096 conv=notruc,noerror >
> slowdrive.img
> [faster server] mount slowdrive.img && calc md5sums > list
> [faster server] rsync /mnt/slowdrive/ /newdir/
> [faster server] cd /newdir/ && check md5sums against list
>
> /** end simplification **/
>
Better.

Make sure that the physical move from one machine to the next is documented 
and witnessed, preferrably by that suspicious guy who otherwise would be the 
one asking all the questions <g>
> -The slow server is 6 years old -- no cdrom, no floppy :-)
> -Data corruption is totally unacceptable.
Question : How much of this data is there?  the entire disk?  Why not install 
a CD(1) and CD/DVD writer(2), boot from a live-CD in (1), mount the hdd 
read-only and write the data to (2)?
You could then run the checksums against the individual files on the old 
drive, write the checksum to another CD, and run the checksum comparisons 
against the CD copies of the data.  This would mean that the copies, if taken 
onto ReadOnly media, are automatically protected from being amended during 
the checking.
You coul do the same procedure mounting the old data drives (read only of 
course!) in a newer system already equipped with a CD writer, but with NO OS 
drives ... run the new system from a live-CD.  This way the OS used to make 
the copies (the liveCD) is also available to be checked, so you can convince 
people that you weren't running anything in the background to alter the data.
> Questions:
> 1) Theory question: will Option 2 produce identical results to Option 1
> only quicker ??
No idea.  Comments above refer to procedure, not to scripting.

> 2) Practical question: is Option 2 "too risky" given the age of  the drives
> ?? I have this fear of possibly harming a hard drive no matter how careful
> .. is my fear justified or baseless ??

Put the old drives on a cable on their own, do not change the jumper status.  
ie connect a slave drive to the slave end of a drive cable with no master. 
etc etc. 
I'm always moving drives.  I have a drawer full of old drives.  I tend to use 
them as "I might need this data one day, but probably not" storage.  I've 
not, so far, had a problem.
Any time I upgrade a system, I copy all the user data to one of these drives, 
label it and stuff the old drive away.  Thats in addition to all the normal 
backup and restore procedures.

Its just a safety net, but it has come in useful several times ... I hadn't 
heard from an old friend for years, had deleted all her contact details from 
all 'live' systems, then was told she was back in the country.
Dug out an old drive, got her mobile phone number, and took her out for a meal 
last night in fact!

> 3) The better idea is to ... I'm always up for the better idea !!!

Everything depends on just what this procedure is is designed to do, and 
whether your setup will satisfy other people, if there are others involved.

1] Run a checksum on the entire drive, mirror it to a new hdd/CD (not a 
mounted partition), then run a checksum on the mirror image.  Compare.  
2] If/when you have to mount the drive with the old data on it, mount it 
read-only.  Log all commands relating to that data automatically, so others 
can 'see' each step.  

> Thanks again and best wishes to everyone.
>
> Andrew

-- 
Tony Dietrich
PGP Key at hkp://wwwkeys.pgp.net
ID D4055CE3


[Index of Archives]     [Current Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [Yosemite Photos]     [KDE Users]     [Fedora Tools]     [Fedora Docs]

  Powered by Linux