* Sander ([email protected]) wrote:
> Dr. David Alan Gilbert wrote (ao):
> > I was using rsync, but the problem with rsync is that I have
> > a back up server then filled with lots and lots of small files
> > - I want larger files for spooling to tape.
> > (Other suggestions welcome)
>
> Can't you just tar the small files from the backupserver to tape? (or,
> what is the problem with that?).
Lots of small files->slow; it is an LTO-2 tape drive that is spec'd
at 35MByte/s - it won't get that if I'm feeding it from something
seeking all over.
> > write a lot more data.
>
> You also do incremental backups?
I could - but they are a pain at restore time.
> > I've benchmarked write performance on the filesystem at
> > 60-70MB/s for a single write process (as shown with iostat)
> > for a simple dd if=/dev/zero of=abigfile bs=1024k
> >
> > My problem is that with the parallel writes iostat is showing
> > I'm actually getting ~3MB/s write bandwidth - that stinks!
>
> How many parallel streams can the system currently handle before the
> write bandwith gets unacceptable?
I'll be honest I don't know; this was running with 9 streams; but
I know the overall speed of the backup goes up as I increase
the parallelism from 5 through 9 - but it still sucks.
> > The machine is a dual xeon with 1GB of RAM, an intel GigE
> > card and a 2.6.11 kernel, a 3ware-9000 series pci-x controller
> > with a 1.5TB RAID5 partition running Reiser3.
>
> What mount options? And how many disks?
7 active discs, raid5; mounted with noatime, nodiratime
> > Reiser3 is used because I couldn't get ext3 stable on a filesystem of
> > this size (-64ZByte free shown in df),
>
> That is not a sign of instability per se AFAIK.
When I fsck it fixes it - this to me is an indication something is wrong
with the ondisc data; now it might only be the freespace totals - but
the fact that the disc contents are wrong makes me worry - I don't
like having to fsck a 1.5TB partition.
> > and xfs didn't seem stable on recovering from an arbitrarily placed
> > reset. The 3ware has write caching (with battery backup).
>
> How is the cache configured in the bios?
Write cache is on in the 3ware bios as is the battery backup.
> > I'm open for all suggestions.
>
> Would it be possible to test software raid to see if that gives
> different numbers?
Erm I guess I could - but the controller does manage
60/70MB/s write as a raw stream, so as far as I can tell if I can
persuade the kernel not to chop my writes into silly small
chunks things should be good.
Dave
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
\ _________________________|_____ http://www.treblig.org |_______/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
| |
 |