large file corruption (x86_64, ext3) under 2.6.17 kernels?

["Gary Funck" <gary@intrepid.com>]

Background: Back in May we installed Fedora Core 5 on a newly acquired
Opteron 275 (dual core, dual cpu) development server with 4G of memory
and a Tyan motherboard.  We used an online backup strategy that we
already had in place, where backups were written to two 300G Maxtor SATA
hard drives.  Recently, while trying to restore files from one of
the incremental backups we noticed that the gzip'd dump file was
corrupted.  We then ran "gunzip -t" on all our backups and discovered
that probably 1/3 of them were corrupted, mainly the larger, level 0
dumps, but not exclusively the level 0 dumps.

The file corruption problem seem to occur on large files,
generally in the 5G to 50G range, though there was one 700M file that
was also corrupted.  We have only about six weeks of backups online
-- we noticed that the corrupted files went back at least that far,
but haven't bothered to pull tapes (which are made weekly from the
images saved on the hard drives) to see if the problem has been
in place ever since we switched to the Opteron server.  Note that
the file systems where we saved the backup files are ext3,
with the -T largefile4 setting, but other than that the settings
were default.

Through the process of elimination we determined that the problem
was not the hard drives, not the SATA controller, not "dump", "dd",
or "gzip".  We know that everything was working fine on Fedora Core 5
on a vanilla x86 platform, prior to the move to the Opteron based
system.  This leaves mainly only the kernel and/or the ext3
file system manager as possible sources of the problem.  Note that
server and kernel aren't showing any other signs of problems: no
unusual meeasges in the system log, no other indications of errors
or problems.

The following command illustrates one of the experiments that we ran:
   dump -0 -b 32 -u -a -f - / | gzip -4 | tee /bak1/dump.0.gz >
/raid0/dump.0.gz
After execution of the command above we would expect that /bak1/dump.0.gz
and /raid0/dump.0.gz are identical.  But many times they were not.  In
the example above, the copy on /raid0 was corrupted (as reported by gunzip
-t),
but the copy on /bak1 was not [/raid0 is a 3-ware controller with two 200G
drives running in a RAID-1 configuraion, and /bak1 is an SATA drive].
We wrote a simple test program that writes sequences of blocks of 32k bytes
each, whith a sequence number in each block.  The test program would
read back the blocks and report errors.  We would see a range in the number
of blocks in error, usually in the low 20's out a several gigabyte file
that had roughly 250,000 32k blocks on it.  The file sizes of both
copies of the file were equal.  We could see no particular pattern in
the data blocks that were in error (such as shifting of the data or
interchange of the blocks).

Up until now, we've been running the stock FC5 kernels:
2.6.17-1.2145_FC5 and 2.6.17-1.2157_FC5, and their recently released
predecessors.  We have reasons to believe that both versions of
the kernel were demonstrating the file corruption that we were
seeing.  I should add that we did reboot several times during our
testing to eliminate the possibility that some system state had
been clobbered.  We found corrupted files on the SATA drives, on
different SATA controllers and on the 3ware.  There was no partiuclar
pattern except that we saw file corruption when writing large files.

Today we built and booted the 2.6.18-rc6 kernel, and re-ran our
tests (twice).  We're not seeing the file corruption errors.
We'll be running more extensive tests overnight.

I looked through the archives and didn't see any mention of
this sort of problem, so am wondering if seomthing like this
has been reported against the 2.6.17 kernel, and/or
whether there are some known fixes in the 2.6.18 kernel that
might have addressed issues like this one?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/