bug in __vm_enough_memory()

[Simon Derr <Simon.Derr@bull.net>]

Hi all,

We have found what seems to be a small bug in __vm_enough_memory()
when sysctl_overcommit_memory is set to OVERCOMMIT_NEVER.

When this bug occurs the systems fails to boot, with /sbin/init whining 
about fork() returning ENOMEM.

We hunted down the problem to this:

The deferred update mecanism used in vm_acct_memory(), on a SMP system, 
allows the vm_committed_space counter to have a negative value.

This should not be a problem since this counter is known to be inaccurate.

But in __vm_enough_memory() this counter is compared to the `allowed' 
variable, which is an unsigned long. This comparison is broken since it 
will consider the negative values of vm_committed_space to be huge 
positive values, resulting in a memory allocation failure.

A proposed fix is attached below.

Signed-off-by: Jean-Marc.Saffroy@ext.bull.net
Signed-off-by: Simon.Derr@bull.net


Index: linux-2.6.12/mm/mmap.c
===================================================================
--- linux-2.6.12.orig/mm/mmap.c	2005-08-02 15:45:30.000000000 +0200
+++ linux-2.6.12/mm/mmap.c	2005-08-02 16:28:48.289575957 +0200
@@ -144,7 +144,10 @@ int __vm_enough_memory(long pages, int c
 	   leave 3% of the size of this process for other processes */
 	allowed -= current->mm->total_vm / 32;
 
-	if (atomic_read(&vm_committed_space) < allowed)
+	/* cast `allowed' as a signed long because vm_committed_space 
+	 * sometimes has a negative value
+	 */
+	if (atomic_read(&vm_committed_space) < (long)allowed)
 		return 0;
 
 	vm_unacct_memory(pages);
Index: linux-2.6.12/mm/nommu.c
===================================================================
--- linux-2.6.12.orig/mm/nommu.c	2005-06-17 21:48:29.000000000 +0200
+++ linux-2.6.12/mm/nommu.c	2005-08-02 16:28:46.384302543 +0200
@@ -1167,7 +1167,10 @@ int __vm_enough_memory(long pages, int c
 	   leave 3% of the size of this process for other processes */
 	allowed -= current->mm->total_vm / 32;
 
-	if (atomic_read(&vm_committed_space) < allowed)
+	/* cast `allowed' as a signed long because vm_committed_space 
+	 * sometimes has a negative value
+	 */
+	if (atomic_read(&vm_committed_space) < (long)allowed)
 		return 0;
 
 	vm_unacct_memory(pages);
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/