possible bug in kmem_cache related code

[Or Gerlitz <ogerlitz@voltaire.com>]

With 2.6.17-rc3 I'm running into something which seems as a bug related 
to kmem_cache. Doing some allocations/deallocations from a kmem_cache and 
later attempting to destroy it yields the following message and trace

============================================================================
slab error in kmem_cache_destroy(): cache `my_cache': Can't free all objects

Call Trace: <ffffffff8106e46b>{kmem_cache_destroy+150}
       <ffffffff88204033>{:my_kcache:kcache_cleanup_module+51}
       <ffffffff81044cd3>{sys_delete_module+415} <ffffffff8112fb5b>{__up_write+20}
       <ffffffff8105d42b>{sys_munmap+91} <ffffffff8100966a>{system_call+126}

Failed to destroy cache
============================================================================

I was hitting it as an Infiniband/iSCSI user as IB/iSCSI/SCSI code use 
kmem_caches, but since the failure happens on a code which works fine on 
2.6.16 i have decided to try it with a synthetic module and had this hit...

Below is a sample code that reproduces it, if i only do kmem_cache_create 
and later destroy it does not happen, attached is my .config please note
that some of the CONFIG_DEBUG_ options are open.

Please CC openib-general@openib.org at least with the resolution of the 
matter since it kind of hard to do testing over 2.6.17-rcX with this 
issue, the tests run fine but some modules are crashing on rmmod so a
reboot it needed...

thanks,

Or.

This is the related slab info line once the module is loaded

my_cache  256    264    328   12    1 : tunables   32   16    8 
: slabdata     22     22      0 : globalstat     264    264    22    0

--- /deb/null	1970-01-01 02:00:00.000000000 +0200
+++ kcache/kcache.c	2006-04-27 10:43:18.000000000 +0300
@@ -0,0 +1,61 @@
+#include <linux/module.h>
+#include <linux/slab.h>
+
+kmem_cache_t *cache;
+
+struct foo {
+	char bar[300];
+};
+
+
+#define TRIES 256
+
+struct foo *foo_arr[TRIES];
+
+static int __init kcache_init_module(void)
+{
+	int i, j;
+
+	cache = kmem_cache_create("my_cache",
+				  sizeof (struct foo),
+				  0,
+				  SLAB_HWCACHE_ALIGN,
+				  NULL,
+				  NULL);
+	if (!cache) {
+		printk(KERN_ERR "couldn't create cache\n");
+		goto error1;
+	}
+
+	for (i = 0; i < TRIES; i++) {
+		foo_arr[i] = kmem_cache_alloc(cache, GFP_KERNEL);
+		if (foo_arr[i] == NULL) {
+			printk(KERN_ERR "couldn't allocate from cache\n");
+			goto error2;
+		}
+	}
+
+	return 0;
+error2:
+	for (j = 0; j < i; j++)
+		kmem_cache_free(cache, foo_arr[j]);
+error1:
+	return -ENOMEM;
+}
+
+static void __exit kcache_cleanup_module(void)
+{
+	int i;
+
+	for (i = 0; i < TRIES; i++)
+		kmem_cache_free(cache, foo_arr[i]);
+
+	if (kmem_cache_destroy(cache)) {
+		printk(KERN_DEBUG "Failed to destroy cache\n");
+	}
+}
+
+MODULE_LICENSE("GPL");
+
+module_init(kcache_init_module);
+module_exit(kcache_cleanup_module);

Attachment: kmem-cache-config.bz2
Description: BZip2 compressed data