On Fri, 3 Nov 2006, Jens Axboe wrote:
On Fri, Nov 03 2006, Brent Baccala wrote:
Hello -
I'm running 2.6.15 (Debian) on a Pentium M laptop, PCI attached ext3
filesystem.
I'm writing my first asynchronous I/O program, and for a while I
thought I was really doing something wrong, but more and more I'm
starting to conclude that the problem might be in the kernel.
Basically, I've narrowed things down to a test program which opens a
large (700 MB) file in O_DIRECT mode and fires off 100 one MB async
reads for the first 100 MB of data. The enqueues take about 5 seconds
to complete, which is also about the amount of time this disk needs to
read 100 MB, so I suspect that it's blocking.
I've gotten the POSIX AIO interface at least tolerably running using
the GLIBC thread-based implementation, but I really want the native
interface working.
I whittled the test program down to use system calls instead of the
POSIX AIO library, and I'm attaching a copy. You put a big file at
'testfile' (it just reads it) and run the program:
baccala@debian ~/src/endgame$ time ./testaio
Enqueues starting
Enqueues complete
real 0m5.327s
user 0m0.004s
sys 0m0.740s
baccala@debian ~/src/endgame$
Of that five seconds, it's almost all spent between the two "enqueues"
messages.
You don't mention what hardware you are running this on (the disk sub
system). io_submit() will block, if you run out of block layer requests.
We have 128 of those by default, but if your io ends up getting chopped
into somewhat smaller bits than 1MiB each, then you end up having to
block on allocation of those. So lets say your /src is mounted on
/dev/sdaX, try:
# echo 512 > /sys/block/sda/queue/nr_requests
(substitute sda for whatever device your /src is on)
and re-test. The time between starting and complete should be a lot
smaller, now that you are not blocking on blkdev request allocation. You
may also want to look at the max_sectors_kb in the queue/ directory,
that'll tell you how large a single io will be at most once it reaches
the driver.
--
Jens Axboe
OK, good question. Here's what the kernel reports about the controller:
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ALI15X3: IDE controller at PCI slot 0000:00:04.0
ACPI: PCI Interrupt 0000:00:04.0[A]: no GSI
ALI15X3: chipset revision 195
ALI15X3: not 100%% native mode: will probe irqs later
ide0: BM-DMA at 0xeff0-0xeff7, BIOS settings: hda:DMA, hdb:pio
ALI15X3: simplex device: DMA forced
ide1: BM-DMA at 0xeff8-0xefff, BIOS settings: hdc:DMA, hdd:DMA
Your suggestion definitely helped! /proc/sys/block/hda/queue/max_sectors_kb
reported 128, so I tried:
# echo 1024 > /sys/block/hda/queue/nr_requests
And am now definately seeing async behavior!
The enqueues still take a noticable amount of time, though, just a lot
less than before. They average 1 second total. That's 100 one-MB
reads, broken down into 128 KB blocks, I suppose, for a total of 800
low-level reads. Setting nr_requests higher (2048) doesn't seem to do
any more good.
I can see that you've put me on the right track, but I am still
puzzling... any idea what the remaining second is being used for?
-bwb
Brent Baccala
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
