Dr. David Alan Gilbert wrote (ao):
> I've got a backup system that I'm trying to eek some more performance
> out of - and I don't understand the right way to get the kernel to
> do disc writes efficiently - and hence would like some advice.
>
> I was using rsync, but the problem with rsync is that I have
> a back up server then filled with lots and lots of small files
> - I want larger files for spooling to tape.
> (Other suggestions welcome)
Can't you just tar the small files from the backupserver to tape? (or,
what is the problem with that?).
> So I'm trying switching to streaming gzip'd tars from each
> client to backup to the server. I have one server that
> opens connections to each of the clients and sucks the data
> using netcat (now netcat6 in ipv4 mode) and writes it to
> disc, one file per client. Now the downside here
> relative to rsync is that it is going to transfer and
> write a lot more data.
You also do incremental backups?
> Now the clients are on 100Mb/s, and the server on GigE,
> the clients sometime have to think while they gzip their data, so I'd
> like to suck data from multiple clients at once. So I run multiple of
> these netcat's in parallel - currently about 9.
>
> I've benchmarked write performance on the filesystem at
> 60-70MB/s for a single write process (as shown with iostat)
> for a simple dd if=/dev/zero of=abigfile bs=1024k
>
> My problem is that with the parallel writes iostat is showing
> I'm actually getting ~3MB/s write bandwidth - that stinks!
How many parallel streams can the system currently handle before the
write bandwith gets unacceptable?
> The machine is a dual xeon with 1GB of RAM, an intel GigE
> card and a 2.6.11 kernel, a 3ware-9000 series pci-x controller
> with a 1.5TB RAID5 partition running Reiser3.
What mount options? And how many disks?
> Reiser3 is used because I couldn't get ext3 stable on a filesystem of
> this size (-64ZByte free shown in df),
That is not a sign of instability per se AFAIK.
> and xfs didn't seem stable on recovering from an arbitrarily placed
> reset. The 3ware has write caching (with battery backup).
How is the cache configured in the bios?
> I'm open for all suggestions.
Would it be possible to test software raid to see if that gives
different numbers?
Sander
--
Humilis IT Services and Solutions
http://www.humilis.net
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
| |
 |