I've got a script I wrote up which backs up my computer using rsync. A
short overview is that I created a file system image using dd, and now I
mount the image and rsync from the server's HD to the image file.
Periodically during large backups, I get the following error:
rsync: connection unexpectedly closed (16931 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(453) [sender=2.6.9]
When I Google for the problem, I get many results about a SSH connection
timing out because of inactivity during the delete stage, but I'm not
using SSH, so I don't get why I would have this problem: both the source
and destination are local.
Any ideas on how to debug this would be great. I've thought of using
strace, but it'd result in a huge file which would have to be sorted
through (as rsync isn't technically crashing, rsync would continue with
a shutdown procedure and the logging wouldn't stop right at the point of
failure).
Attached is the backup script I use (the configuration file it loads in
is very simple, so I'm not including that).
Justin W
#!/bin/bash
# User where the reports of the backup are sent to
MAILTO=root
# Path to backup's configuration file
CONF_PATH=/root
# Overview of backup process
# 1) Read and parse configuration file
# 2) Loop through each backup
# 3) Create a FS snapshot using LVM
# 4) Mount the snapshot FS
# 5) Mount our backup file in a loop filesystem (we can guarantee the space
# will be present this way, and it allows selective recovery of files if
# need be)
# 6) rsync the differences between the snapshot FS and the backup FS
# 7) Unmount the backup FS
# 8) Unmount the snapshot FS
# 9) Delete the snapshot LVM (leaving it will degrade performance)
# 10) Repeat 3-9 if necessary
# Give us a large enough path to execute commands
PATH=/bin:/sbin:/usr/bin:/usr/sbin
# Save the old IFS variable so that we can restore it when we're done
ORIGIFS=$IFS
IFS=$(echo -en "\n\b")
[ -f "$CONF_PATH/backup.conf" ] || {
echo "Configuration file not found ($CONF_PATH/backup.conf). Exiting.";
exit 1; }
# Read from the configuration file any exclude filters for the rsync output
LOG_EXCLUDE=$(cat "$CONF_PATH/backup.conf" | egrep "^[ \t]*LOG_EXCLUDE=" |
cut -d'=' -f 2-)
LOG_FILE=/tmp/backup$$.log
# Get each line from the configuration file one at a time, ignoring anything
# that starts with a pound (#) sign
for i in $(cat "$CONF_PATH/backup.conf" | egrep -v "^[ \t]*#.*|^[ \t]*LOG_EXCLUDE=.*"); do
# Parse out the locations we need
VOL=$(echo $i | sed "s@^\([^:]*\):\(.*\)@\1@")
TARG=$(echo $i | sed "s@^\([^:]*\):\(.*\)@\2@")
# Check to make sure the paths are valid
[ -b $VOL ] || { echo "Source volume not valid: $VOL. Skipping";
continue; }
[ -f $TARG ] || { echo "Target image not valid: $TARG. Skipping";
continue; }
# Get necessary information about the source volume.
# Get only the necessary line, and not the headers, and then compress the
# info so it is only separated by a single space (makes finding fields
# easier in the next steps). There will be an extra space at the beginning,
# so fields are 1 greater than what they seem.
#
# Make sure the volume is a LVM Logical Volume
lvs -a $VOL 2>/dev/null 1>/dev/null || {
echo "$VOL not an LVM Logical Volume ($?). Skipping."; continue; }
# Now we actually collect the info
INFO=$(lvs -a $VOL | tail -n 1 | sed 's/[ ][ ]*/ /g')
# Field 1: Logical Volume Name
VOL_NAME=$(echo $INFO | cut -d' ' -f 2)
# Field 2: Volume Group Name
GROUP_NAME=$(echo $INFO | cut -d' ' -f 3)
# Field 3: Attributes - Not needed right now, but may come in handy
ATTRS=$(echo $INFO | cut -d' ' -f 4)
# Field 4: Volume Size
VOL_SIZE=$(echo $INFO | cut -d' ' -f 5)
# Now we just create a name for the snapshot volume, and then we can
# create it. Use a unique name guaranteed to be the same through multiple
# runs (or else blkid.tab fills with unneccessary partition info)
SNAP_NAME=backup_$(echo "$GROUP_NAME/$VOL_NAME" | md5sum | cut -c-32)
SNAP_PATH=/dev/${GROUP_NAME}/${SNAP_NAME}
lvcreate -s -L${VOL_SIZE} -n $SNAP_NAME $VOL >/dev/null || {
echo "Snapshot of '${GROUP_NAME}/${SNAP_NAME}' failed ($?). Skipping.";
continue; }
# Mount our filesystems at temporary mount locations
SNAP_MOUNT=/tmp/$SNAP_NAME
TARG_MOUNT=/tmp/target-$SNAP_NAME
mkdir $SNAP_MOUNT || {
echo "Failed to create snapshot mount point ($?). Skipping.";
cleanup; continue; }
mkdir $TARG_MOUNT || {
echo "Failed to create target mount point ($?). Skipping.";
cleanup; continue; }
mount $SNAP_PATH $SNAP_MOUNT || {
echo "Mounting snapshot failed ($?). Skipping.";
cleanup; continue; }
mount -o loop $TARG $TARG_MOUNT || {
echo "Mounting target image failed ($?). Skipping.";
cleanup; continue; }
# Update the backup file system with files from the snapshot
# IMPORTANT!!: The trailing slash on the source is necessary for it to
# work in the expected way, or else all backups will end up within a
# subdirectory with the name $SNAP_MOUNT
strace -o /tmp/backup-fs \
rsync -aHEAXxv --stats --delete --ignore-errors $SNAP_MOUNT/ $TARG_MOUNT | \
egrep -v "$LOG_EXCLUDE" >> $LOG_FILE
# Cleanup stuff is put into a function so that it can be called before
# an exit call
function cleanup() {
# Unmount the backups and remove the temporary dirs
# Errors may occur here, but we don't really care. Just pipe the errors
# to /dev/null
umount $SNAP_MOUNT 2>/dev/null 1>/dev/null
umount $TARG_MOUNT 2>/dev/null 1>/dev/null
[ -d $SNAP_MOUNT ] && rmdir $SNAP_MOUNT
[ -d $TARG_MOUNT ] && rmdir $TARG_MOUNT
# Delete the snapshot
[ -b $SNAP_PATH ] && lvremove -f $SNAP_PATH >/dev/null
}
# Now call our cleanup routine on normal completions
cleanup
done
if [ -f $LOG_FILE ]; then
# Mail the log file, and then delete the file
mail -s "Backups on $(hostname)" $MAILTO < $LOG_FILE
rm $LOG_FILE
fi
# Restore the IFS
IFS=$ORIGIFS
