Once upon a time, Les Mikesell <lesmikesell@xxxxxxxxx> said: > On Thu, 2006-12-21 at 08:14 -0600, Chris Adams wrote: > > The fact that Linux stops sending on the network sometimes and stops > > reading the hard drive other times points directly to how the kernel is > > caching writes to NFS (but I can't tell if it is the filesystem layer or > > the network stack). > > I think it is access to the physical hard drive, not particularly > NFS that is your bottleneck and it is related to normal buffered > writes. You'll see about the same pattern if you dd between two > local hard drives. If your nfs mount has the sync option it might > make things worse. No, because on both ends, the hard drive goes idle. If I test the speed of the hard drives (using bonnie++), I get higher rates. I did some more testing. a) If I read from the client hard drive with ntfsclone and write to an NFS filesystem (across the gigE link), I get about 200Mbps. b) If I bypass NFS and just use netcat on each end to send/receive data (still reading from the client hard drive with ntfsclone and writing to the server hard drive by just redirecting the output of nc), I get about 278Mbps. c) If I run the same test on the client but just dump to /dev/null on the client (no network involved), the hard drive LED is on continuously and I get about 312Mbps. d) If I dd from /dev/zero on the client (using bs=1024), pipe to netcat, and write to /dev/null on the server, I get about 343Mbps. So, it would appear that NFS is my biggest choke point, but there is still some buffering issue between reading from the hard drive and writing to the network. I tried doing the dd test on another client system, and I got up to 894Mbps (using jumbo frames gets it up to about 971Mbps), so I guess part of my problem is the network interface in the first client (Broadcom NetXtreme BCM5751M); the server is a Broadcom NetXtreme BCM5702X and the second client is a Broadcom NetXtreme BCM5704. -- Chris Adams <cmadams@xxxxxxxxxx> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
