James Wilkinson wrote:
James Foris wrote:
I have run into an issue with memory bandwidth using the Fedora Core
2 kernels and I need help. I don't know what is wrong, but something
killed performance of my custom driver when I ported it from RedHat 7.3
to Fedora Core 2. I believe I have narrowed it down to the kernel.
My driver requires a large amount of contiguous physical memory for
DMA from a PCI device. I use the 'mem=YYY' command line parameter to
reserve the top of physical RAM for my driver. Then I allow mapping
via mmap() calls to user space. The user space app then uses this
pointer to save the data to disk.
<snip>
I have tried this on several platforms and kernels and the results vary,
but the common denominator seems to be:
Fedora kernel + 32-bit Intel = poor performance (see below)
I suggested:
Wild guess: have you tried turning off the 4G+4G support in the kernel
(requires a recompile)?
James replied
I have rebuilt now w/o several patches; 4Kstacks, netdump, and 4g/4g.
The problem has gone away. I suspect that 4G/4G is broken.
It sounds more "broken as designed", to be honest. It sounds as though
your setup is doing a *lot* of context switches between user mode and
kernel mode. The basic trade-off that 4G/4G flushes the TLB each
context switch, so that kernel and user both get nearly 4 GB, and TLB
flushes are expensive.
See
http://lwn.net/Articles/39283/
and
http://kerneltrap.org/node/view/2891
May I recommend fedora-devel-list@xxxxxxxxxx or the LKML? You might not
have much done about it, but they might be able to recommend
workarounds.
Actually, we have gotten some of the kernel group at Intel into the game, now.
It seems that the problem can be completely reproduced using the native
"/dev/mem" interface - no custom drivers needed. The intel engineer that
has looked at this believes that either the 4g/4g patch is broken, or that there
are mem driver interactions with 4g/4g that no one has taken into account
(which to me says that potentially both the mem driver and 4g/4g patch need
fixing).
Certainly, removing the 4g/4g patch fixes this problem.
We have been told that Intel will be talking/working with the Red Hat people
directly to resolve this. They seem to think that this is something significant
that should be resolved.
At the very least, it would be good if real-life problems with 4G/4G are
reported.
James.
