How to diagnose a kernel memory leak

[Bruce Guenter <bruceg@em.ca>]

Greetings.

I am trying to diagnose a slow kernel memory leak, and am having no luck
in pining it down.

I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the
same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4.  Over the
course of several days, the server in question has the amount of
available memory (free minus buffers+cache) gradually decrease.  If I
leave it go, it does eventually thrash itself to death after about a
week (give or take).  The rate is about 150MB per day (the system has
2GB of RAM total so it takes several days).  The working set of
processes remains the same through the whole period at between 50-150MB
(depending on if you count VSZ or RSS).  Nothing shows up in dmesg
except for a couple of one-time lockd and nfs messages  (the system uses
two remote filesystems).  The local filesystems are ReiserFS on a 3Ware
7500-4 controller, and the NIC is an Intel E100.

# free
             total       used       free     shared    buffers     cached
Mem:       2076180    2024068      52112          0     166760      93200
-/+ buffers/cache:    1764108     312072
Swap:      1028152         56    1028096

# cat /proc/meminfo
MemTotal:      2076180 kB
MemFree:         63080 kB
Buffers:        158776 kB
Cached:          91664 kB
SwapCached:          4 kB
Active:        1055244 kB
Inactive:       874660 kB
HighTotal:     1179072 kB
HighFree:          640 kB
LowTotal:       897108 kB
LowFree:         62440 kB
SwapTotal:     1028152 kB
SwapFree:      1028096 kB
Dirty:             768 kB
Writeback:           0 kB
Mapped:          12648 kB
Slab:            69872 kB
CommitLimit:   2066240 kB
Committed_AS:    26316 kB
PageTables:       1492 kB
VmallocTotal:   114680 kB
VmallocUsed:      4700 kB
VmallocChunk:   109784 kB

I would be happy to provide any additional information.  As it stands, I
have to reboot about once a week to clear the RAM or else it thrashes
itself to death.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

Attachment: pgp6qydtKAYPl.pgp
Description: PGP signature