Rsync fails on local transfers

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


I've got a script I wrote up which backs up my computer using rsync.  A
short overview is that I created a file system image using dd, and now I
mount the image and rsync from the server's HD to the image file.

Periodically during large backups, I get the following error:

    rsync: connection unexpectedly closed (16931 bytes received so far)
    rsync error: error in rsync protocol data stream (code 12) at
    io.c(453) [sender=2.6.9]

When I Google for the problem, I get many results about a SSH connection
timing out because of inactivity during the delete stage, but I'm not
using SSH, so I don't get why I would have this problem: both the source
and destination are local.

Any ideas on how to debug this would be great. I've thought of using
strace, but it'd result in a huge file which would have to be sorted
through (as rsync isn't technically crashing, rsync would continue with
a shutdown procedure and the logging wouldn't stop right at the point of

Attached is the backup script I use (the configuration file it loads in
is very simple, so I'm not including that).

Justin W


# User where the reports of the backup are sent to
# Path to backup's configuration file

# Overview of backup process
#  1) Read and parse configuration file
#  2) Loop through each backup
#  3) Create a FS snapshot using LVM
#  4) Mount the snapshot FS
#  5) Mount our backup file in a loop filesystem (we can guarantee the space
#     will be present this way, and it allows selective recovery of files if
#     need be)
#  6) rsync the differences between the snapshot FS and the backup FS
#  7) Unmount the backup FS
#  8) Unmount the snapshot FS
#  9) Delete the snapshot LVM (leaving it will degrade performance)
# 10) Repeat 3-9 if necessary

# Give us a large enough path to execute commands

# Save the old IFS variable so that we can restore it when we're done
IFS=$(echo -en "\n\b")

[ -f "$CONF_PATH/backup.conf" ] || {
	echo "Configuration file not found ($CONF_PATH/backup.conf). Exiting.";
	exit 1; }

# Read from the configuration file any exclude filters for the rsync output
LOG_EXCLUDE=$(cat "$CONF_PATH/backup.conf" | egrep "^[ \t]*LOG_EXCLUDE=" |
              cut -d'=' -f 2-)

# Get each line from the configuration file one at a time, ignoring anything
# that starts with a pound (#) sign
for i in $(cat "$CONF_PATH/backup.conf" | egrep -v "^[ \t]*#.*|^[ \t]*LOG_EXCLUDE=.*"); do
	# Parse out the locations we need
	VOL=$(echo $i | sed "[email protected]^\([^:]*\):\(.*\)@\[email protected]")
	TARG=$(echo $i | sed "[email protected]^\([^:]*\):\(.*\)@\[email protected]")

	# Check to make sure the paths are valid
	[ -b $VOL ]  || { echo "Source volume not valid: $VOL. Skipping";
	                  continue; }
	[ -f $TARG ] || { echo "Target image not valid: $TARG. Skipping";
	                  continue; }

	# Get necessary information about the source volume.
	# Get only the necessary line, and not the headers, and then compress the
	# info so it is only separated by a single space (makes finding fields
	# easier in the next steps). There will be an extra space at the beginning, 
	# so fields are 1 greater than what they seem.
	# Make sure the volume is a LVM Logical Volume
	lvs -a $VOL 2>/dev/null 1>/dev/null || {
		echo "$VOL not an LVM Logical Volume ($?). Skipping."; continue; }
	# Now we actually collect the info
	INFO=$(lvs -a $VOL | tail -n 1 | sed 's/[ ][ ]*/ /g')
	# Field 1: Logical Volume Name
	VOL_NAME=$(echo $INFO | cut -d' ' -f 2)
	# Field 2: Volume Group Name
	GROUP_NAME=$(echo $INFO | cut -d' ' -f 3)
	# Field 3: Attributes - Not needed right now, but may come in handy
	ATTRS=$(echo $INFO | cut -d' ' -f 4)
	# Field 4: Volume Size
	VOL_SIZE=$(echo $INFO | cut -d' ' -f 5)

	# Now we just create a name for the snapshot volume, and then we can
	# create it. Use a unique name guaranteed to be the same through multiple
	# runs (or else fills with unneccessary partition info)
	SNAP_NAME=backup_$(echo "$GROUP_NAME/$VOL_NAME" | md5sum | cut -c-32)
	lvcreate -s -L${VOL_SIZE} -n $SNAP_NAME $VOL >/dev/null || {
		echo "Snapshot of '${GROUP_NAME}/${SNAP_NAME}' failed ($?). Skipping.";
		continue; }
	# Mount our filesystems at temporary mount locations
	mkdir $SNAP_MOUNT || {
		echo "Failed to create snapshot mount point ($?). Skipping.";
		cleanup; continue; }
	mkdir $TARG_MOUNT || {
		echo "Failed to create target mount point ($?). Skipping.";
		cleanup; continue; }
	mount $SNAP_PATH $SNAP_MOUNT || {
		echo "Mounting snapshot failed ($?). Skipping.";
		cleanup; continue; }
	mount -o loop $TARG $TARG_MOUNT || {
		echo "Mounting target image failed ($?). Skipping.";
		cleanup; continue; }

	# Update the backup file system with files from the snapshot
	# IMPORTANT!!: The trailing slash on the source is necessary for it to
	# work in the expected way, or else all backups will end up within a
	# subdirectory with the name $SNAP_MOUNT
	strace -o /tmp/backup-fs \
	rsync -aHEAXxv --stats --delete --ignore-errors $SNAP_MOUNT/ $TARG_MOUNT | \
		egrep -v "$LOG_EXCLUDE" >> $LOG_FILE

	# Cleanup stuff is put into a function so that it can be called before
	# an exit call
	function cleanup() {
		# Unmount the backups and remove the temporary dirs
		# Errors may occur here, but we don't really care. Just pipe the errors
		# to /dev/null
		umount $SNAP_MOUNT 2>/dev/null 1>/dev/null
		umount $TARG_MOUNT 2>/dev/null 1>/dev/null
		[ -d $SNAP_MOUNT ] && rmdir $SNAP_MOUNT
		[ -d $TARG_MOUNT ] && rmdir $TARG_MOUNT

		# Delete the snapshot
		[ -b $SNAP_PATH ] && lvremove -f $SNAP_PATH >/dev/null
	# Now call our cleanup routine on normal completions

if [ -f $LOG_FILE ]; then
	# Mail the log file, and then delete the file
	mail -s "Backups on $(hostname)" $MAILTO < $LOG_FILE

# Restore the IFS

[Index of Archives]     [Current Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [Yosemite Photos]     [KDE Users]     [Fedora Tools]     [Fedora Docs]

  Powered by Linux