Re: ZONE_NORMAL memory exhausted by 4000 TCP sockets

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



2006/11/6, Eric Dumazet <[email protected]>:
Zhao Xiaoming a écrit :
> Dears,
>    I'm running a linux box with kernel version 2.6.16. The hardware
> has 2 Woodcrest Xeon CPUs (2 cores each) and 4G RAM. The NIC cards is
> Intel 82571 on PCI-e bus.
>    The box is acting as ethernet bridge between 2 Gigabit Ethernets.
> By configuring ebtables and iptables, an application is running as TCP
> proxy which will intercept all TCP connections requests from the
> network and setup another TCP connection to the acture server.  The
> TCP proxy then relays all traffics in both directions.
>    The problem is the memory. Since the box must support thousands of
> concurrent connections, I know the memory size of ZONE_NORMAL would be
> a bottleneck as TCP packets would need many buffers. After setting
> upper limit of net.ipv4.tcp_rmem and net.ipv4.tcp_wmem to 32K bytes,
> our test began.
>    My test scenario employs 2000 concurrent downloading connections
> to a IIS server's port 80. The throughput is about 500~600 Mbps which
> is limited by the capability of the client application. Because all
> traffics are from server to client and the capability of client
> machine is bottleneck, I believe the receiver side of the sockets
> connected with server and the sender side of the sockets connected
> with client should be filled with packets in correspondent windows.
> Thus, roughly there should be about 32K * 2000+ 32K*2000 = 128M bytes
> memory occupied by TCP/IP stack for packet buffering. Data from
> slabtop confermed it. it's about 140M bytes memory cost after I start
> the traffic. That reasonablly matched with my estimation. However,
> /proc/meminfo had a different story. The 'LowFree' dropped from about
> 710M to 80M. In other words, there's addtional 500M memory in
> ZONE_NORMAL allocated by someone other than the slab. Why?

We dont know. You might post some data so that we can have some ideas.

Also, these kind of question is better handled by linux netdev mailing list,
so I added a CC to this list.

cat /proc/slabinfo
cat /proc/meminfo
cat /proc/net/sockstat
cat /proc/buddyinfo

>   I also made another test that the upper limit of tcp_rmem and
> tcp_wmem being set to 64K. After 2000 connections transfering a lot of
> data for several seconds, the linux box showed some error messages
> such as error allocating memory pages, etc. and became unstable.
>   My questions are:
>
> 1. To calculate memory request of TCP sockets, is there any other
> large amount of memory requested besides send and receive buffer?
> 2. Is there any logics that emploied by TCP/IP stack that will
> dynamically allocating memory pages directly instead of from slab?

TCP stack is one thing, but other things may consume ram on your kernel.

Also, kernel memory allocation might use twice the ram you intend to use
because of power of two alignments.

Are you using iptables connection tracking ?

If you plan to use a lot of RAM in kernel, why dont you use a 64 bits kernel,
so that all ram is available for kernel, not only 900 MB ?

Eric


Thanks for the answer. I know it's more likely relats to netdev.
However, it's always a strange thing to have 400~500M bytes LOWMEM
'gone' while it's not reported to be occupied by slab. Both meminfo
and buddyinfo tell the same.

with traffics of 2000 concurrent sessions:

cat /proc/meminfo
MemTotal:      4136580 kB
MemFree:       3298460 kB
Buffers:          4096 kB
Cached:          21124 kB
SwapCached:          0 kB
Active:          47416 kB
Inactive:        12532 kB
HighTotal:     3276160 kB
HighFree:      3214592 kB
LowTotal:       860420 kB
LowFree:         83868 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:              12 kB
Writeback:           0 kB
Mapped:          42104 kB
Slab:           293952 kB
CommitLimit:   2068288 kB
Committed_AS:    58892 kB
PageTables:       1112 kB
VmallocTotal:   116728 kB
VmallocUsed:      2940 kB
VmallocChunk:   110548 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB


without traffics:

cat /proc/meminfo
MemTotal:      4136580 kB
MemFree:       3924276 kB
Buffers:          4460 kB
Cached:          21020 kB
SwapCached:          0 kB
Active:          47592 kB
Inactive:        12848 kB
HighTotal:     3276160 kB
HighFree:      3214716 kB
LowTotal:       860420 kB
LowFree:        709560 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:              40 kB
Writeback:           0 kB
Mapped:          42368 kB
Slab:           132172 kB
CommitLimit:   2068288 kB
Committed_AS:    59220 kB
PageTables:       1140 kB
VmallocTotal:   116728 kB
VmallocUsed:      2940 kB
VmallocChunk:   110548 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB


Xiaoming.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Index of Archives]     [Kernel Newbies]     [Netfilter]     [Bugtraq]     [Photo]     [Stuff]     [Gimp]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Video 4 Linux]     [Linux for the blind]     [Linux Resources]
  Powered by Linux