Re: how would you backup 1TB of data to dvds?

[Dave M <DaveM@xxxxxxxx>]

At 12:46 PM 2/9/2008 , Valent wrote:
> How would you backup 1TB of data in a server with 4x 250GB drives all
> mounted as separate mount points to a set of dvds using tools
> available in fedora, centos or rhel?

It is amazing how much BS this thread spawned without providing an answer 
to your question. I am in a similar situation and here is how I do it.

I use dar to make archive slices of my data and then use the attached 
script to burn and verify each slice to a DVD. The script also creates a 
catalog file (that you can reference for future backups/restores), a parity 
file for each slice and verifies the burn. I use cron to run this on a 
daily basis and e-mail myself when it fails (so I can change the DVD and 
the reference point to use the latest dar catalog file). This works well as 
long as your daily backups will fit on one DVD or less.

Here is the dar command that I use:

#--------------------------------------
#!/bin/bash
# Run this script to create an archive using the following dar command.
dar -v -c "/home/davem/Backup/HomeP/HomeP_diff_"$(date +%y%m%d-%H%M)"" -R 
"/home/" -A "/home/davem/Backup/HomeP/Cat/HomeP_diff_080120-0230" -s 
419430400 -D -y -m 150 -Z "*.*_" -Z "*.aac" -Z "*.avi" -Z "*.bin" -Z 
"*.bz2" -Z "*.cab" -Z "*.deb" -Z "*.dv" -Z "*.exe" -Z "*.gif" -Z "*.gz" -Z 
"*.iso" -Z "*.jpg" -Z "*.lzo" -Z "*.m4a" -Z "*.m4v" -Z "*.mov" -Z "*.mp3" 
-Z "*.mp4" -Z "*.mpg" -Z "*.ogg" -Z "*.oma" -Z "*.par" -Z "*.par2" -Z 
"*.pbm" -Z "*.pdf" -Z "*.png" -Z "*.pnm" -Z "*.rar" -Z "*.rm" -Z "*.rm1" -Z 
"*.rpm" -Z "*.tgz" -Z "*.vmdk" -Z "*.wma" -Z "*.wmv" -Z "*.Z" -Z "*.zip" -E 
"/home/davem/Scripts/KDar_BurnSlice.sh %p %b %n %e >> %p/%b.log 2>&1" -X 
".*~" -X "*~" -X "*.bak" -X "*.bup" -X "*.BUP" -X "*.cue" -X "*.dar" -X 
"*._dd" -X "*.dv" -X "*.ifo" -X "*.IFO" -X "*.tmp" -X "*.vob" -X "*.wav" -P 
"davem/Backup/HomeP"
#------------------------------------------------

Note that the -v option tells dar to be verbose.

The -c option specifies the base name of the archive slice. I use $(date) 
to append the date to the archive slice names.

The -R option tells dar where to find the files to add to my archive.

The -A option points dar to a reference archive (or catalog). Only files 
that have been added or modified since the reference archive was created 
will be added to the new archive.

The -s option tells dar how large each slice should be. I use 400Mb so 10 
slices and their parity files (40Mb each) will fit on a DVD.

The -E option executes the command (in quotes). I run a script that makes a 
parity (par2) file and then burns/verifies each slice/parity file to the 
DVD. The command is "/home/davem/Scripts/KDar_BurnSlice.sh %p %b %n %e >> 
%p/%b.log 2>&1" where %p is the path to the slice, %b is the bas name of 
the slice, %n is the slice number and %e is the slice extension (usually 
.dar).

The -Z option tells dar not to try to compress these files (because they 
are already compressed).

The -m option tells dar not to try to compress files that are smaller than 
the number of bytes specified (since small files do not compress efficiently).

The -P option tells dar to skip the files in the specified directory.

The -X option tells dar to skip the specified files.

The -D option tells dar to store skipped directories as empty directories

Attached is the script that I use to burn/verify each slice to DVD's.

Note that this dar command works by referencing an initial backup. I would 
suggest dividing your data set into smaller chunks, back them up separately 
and then start a new directory for all of your future files. Otherwise you 
will have to get a single complete backup of your whole file set to use as 
a reference for future backups. That will be a long and dicey process given 
that DVD's are so prone to errors.

I would also suggest that you do overlapping backups. Each time I change my 
DVD I do another  backup before referencing the catalog file from the 
previous DVD. The downside is that you fill up a good chunk of the new DVD 
but the benefit is that you have two copies of your data. One copy is in 
the backup sets stored on the previous DVD and a second copy is contained 
in the first backup set on the new DVD. I do this in case one of the DVDs 
is lost, damaged or has errors that can not be recovered using par2. This 
is not necessary but, if your data is valuable, I highly suggest having 
redundant copies.

Hope this helps.

Attachment: KDar_BurnSlice.tar.gz
Description: Binary data

-------------------------------
Dave M
Davem (at) Mich (dot) Com
Ann Arbor, Mich. USA

First Sale doctrine under attack, libraries at risk: 

http://www.eff.org/deeplinks/archives/005400.php