Matthijs van der Klip wrote:
Hi,
I hope someone can enlighten me on the behaviour of malloc with respect to
'Active' memory.
Circumstances:
I have installed FC4 x86_64 on a Dual Opteron machine with 8GB of physical
memory. I am planning to use this box solely for hosting a large MySQL
database, so I configured MySQL 4.1 (official x86_64, glibc2.3 rpm) to use
7GB for it's InnoDB buffer pool (innodb_buffer_pool_size). So far this
works flawlessly.
The problem:
After doing several compiles on the machine (it is my only x86_64 machine
at the moment), some memory is used by the kernel for several uses. Let's
have a look at /proc/meminfo:
MemTotal: 8124080 kB
MemFree: 385772 kB
Buffers: 201632 kB
Cached: 661568 kB
SwapCached: 0 kB
Active: 5958788 kB
Inactive: 1463764 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 8124080 kB
LowFree: 385772 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 270844 kB
Slab: 284044 kB
CommitLimit: 4062040 kB
Committed_AS: 2618264 kB
PageTables: 16772 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 288 kB
VmallocChunk: 34359738043 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
Only roughly 375 MB 'free' memory left. Not a problem by itself, the
kernel is expected to make good use of the memory it can get. However when
I stop MySQL and then try to restart it, I get the following error:
050726 13:56:21 InnoDB: Error: cannot allocate 7340048384 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 78086952 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We keep retrying the allocation for 60 seconds...
InnoDB: Fatal error: cannot allocate the memory for the buffer pool
Even after hours later I cannot reclaim the memory. Only after decreasing
the InnoDB buffer pool to roughly 1.6GB (found this by trial and error),
MySQL can be successfully started again. Now it sound rather ridiculous
that I can use only 1.6GB of 8GB physical memory. Can anybody explain
where the rest of the memory has gone? According to /proc/meminfo output I
would guess it be in the 'Active' region:
'Active: Memory that has been used more recently and usually not reclaimed
unless absolutely necessary.'
(http://www.redhat.com/advice/tips/meminfo.html)
How do I reclaim this memory? I need it for MySQL. The only way I have
found so far is rebooting the machine, but I cannot imagine that would be
the ultimate solution.
Some snippets of MySQL code:
ulint n, /* in: number of bytes to allocate */
ret = malloc(n + sizeof(ut_mem_block_t));
if (ret == NULL && retry_count < 60) {
if (retry_count == 0) {
ut_print_timestamp(stderr);
fprintf(stderr,
" InnoDB: Error: cannot allocate %lu bytes of\n"
Thanks for your time.
The question is what did MySQL do with the memory? If it was left
as a malloc'd region, it should get returned to the free pool by the
garbage collection system but it's not instantaneous. I'd wait a bit,
then check the free space and try again.
If the memory was, instead, given over to some IPC mechansim (shared
memory, unnamed pipes, messages, semaphores, etc.), then it won't be
cleaned up and you'll have to kill those off manually with an ipcrm
command (see "man ipcs", "man ipcrm" for details).
Of course, MySQL should trap all signals and, on receiving a SIGTERM,
SIGHUP, SIGQUIT, etc., go to a routine that releases all of its memory
before it terminates. That's just good practice and something that is
overlooked by many programmers far too often.
----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer rstevens@xxxxxxxxxxxxxxx -
- VitalStream, Inc. http://www.vitalstream.com -
- -
- Millihelen, adj: The amount of beauty required to launch one ship. -
----------------------------------------------------------------------
