what's my kernel doing?

[NetComrade <netcomrade@bookexchange.net>]

We have a 4CPU AMD (v40z) machine running RedHat 4.0 (2.6.9-11.ELsmp), and
I noticed high CPU utilization especially on the system side:

[root@mars ~]# mpstat 5
Linux 2.6.9-11.ELsmp (mars)     04/04/2006

01:37:32 PM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle
intr/s
01:37:37 PM  all   38.50    0.00   50.95    0.00    0.30    3.20    7.05
6750.80
01:37:42 PM  all   41.91    0.00   53.10    0.00    0.30    3.10    1.60
6884.60
01:37:47 PM  all   39.97    0.00   53.08    0.00    0.25    3.05    3.65
6672.65


Initially I thought it was a huge file system cache that system was
scanning a lot. 

[VCRS@mars admin]$free
             total       used       free     shared    buffers     cached
Mem:      16032636   12539720    3492916          0     244440   11640608
-/+ buffers/cache:     654672   15377964
Swap:      8385920         28    8385892

However, I unmounted/umounted the oracle file system in directio mode the
cache didn't go away, so it must be some root disk cache.. (there also
used to be a parameter to control pagecache up to kernel 2.6
(/proc/sys/vm/pagecache), but it seems to be gone now, couldn't find an
equivalent nor the reason it is gone)

Then I installed the 'profiler', which I can barely understand, but
there's a difference between sun & mars (identical machines, both running
oracle databases). It seems weird that some usb process is taking up so
much cpu, however on sun (which doesn't have a similar system utilization)
it's near the 'top' too, just at a much lower percentage.

If anyone here could give me better advice on how to go about this problem
(high system cpu utilization), or point me to some resources, I'd really
appreciate it.


[root@sun ~]# opreport|head
CPU: AMD64 processors, speed 2592.7 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
  1567396 54.8034 oracle
   904729 31.6335 vmlinux
   136014  4.7557 libc-2.3.4.so
    33860  1.1839 e1000
    26277  0.9188 vxfs

[root@sun ~]# opreport -l
/usr/lib/debug/lib/modules/2.6.9-5.EL/vmlinux|head
CPU: AMD64 processors, speed 2592.7 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
samples  %        symbol name
113702   11.3522  usbfs_add_bus
80084     7.9957  get_user_pages
53515     5.3430  init_8259A
40519     4.0455  sys_access
28709     2.8663  load_elf32_binary
26368     2.6326  setscheduler
23848     2.3810  __ide_dma_write


Oh yeah, 2.6.9-5EL was the only 'kernel-debug' rpm I could find.


[root@mars tmp]# opreport|head
CPU: AMD64 processors, speed 2592.66 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
 23361362 66.3184 vmlinux
  9122930 25.8982 oracle
  1102906  3.1309 no-vmlinux
   539234  1.5308 libc-2.3.4.so
   183650  0.5213 e1000

[root@mars tmp]# opreport -l
/usr/lib/debug/lib/modules/2.6.9-5.EL/vmlinux|head
CPU: AMD64 processors, speed 2592.66 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
samples  %        symbol name
8155421  34.4932  usbfs_add_bus
4776837  20.2035  get_user_pages
2954158  12.4945  sys_access
1702339   7.2000  setscheduler
779719    3.2978  sys_fchdir
367546    1.5545  sys_open
260518    1.1019  init_8259A





































Please let me know which list would be appropriate for a question like
this, if this list is for kernel development only.

Thanks,
andrey

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/