-
Title
Out of Socket Memory Error and things to check -
Description
This article is meant to address the Out of Socket Memory in relation to Syslog-NG PE and some things worth looking into. -
Resolution
The following two items are worth investigating.
TCP Memory
You can compare how much memory your kernel is configured to give to TCP vs how much is actually being used. The kernel keeps track of the memory allocated to TCP.
This is done in a multiple of pages, not in bytes. 1 page generally equals 4096 bytes.
Find out how much memory the kernel is giving to TCP.
$ cat /proc/sys/net/ipv4/tcp_mem
3093984 4125312 6187968These values are in 'pages' and are for a machine with 32GB or memory. They represent the following.
- When usage is below 3093984 (11.8GB) this is considered low by the kernel and won't action anything.
- At more than 4125312 (15.7GB) this enters a "memory pressure" mode.
- The max available to TCP is 6197968 (23.6GB) and when this is exceeded it causes "Out of Socket memory" errors.
To find how much is actively being used the following command is useful.
$ cat /proc/net/sockstat
sockets: used 14565
TCP: inuse 35938 orphan 21564 tw 70529 alloc 35942 mem 1894
UDP: inuse 11 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0The second line of output represented by mem is the number of pages allocated to TCP. In this example we are well below even the low threshold.
Too many orphan sockets
An orphan socket is a socket that is not associated to a file descriptor.To find the limit on the number of orphan sockets run the following$ cat /proc/sys/net/ipv4/tcp_max_orphans
65536This examples shows the default value of 64k. To determine the number of current orphan socktes we can use this command.
$ cat /proc/net/sockstat
sockets: used 14565
TCP: inuse 35938 orphan 21564 tw 70529 alloc 35942 mem 1894
[...]21564 orphans appears to be well below the threshold of 65536. However the code that prints the warning has a 'shift' variable that has a value between 0 and 2. The check is (orphans << shift > sysctl_tcp_max_orphans). In certain cases this results in the kernel penalizing some sockets more. It does this by multiplying the number of orphans by two or four times more. This can artificially increase the bad sockets. This can trigger "Out of Socket memory" issues when when you are still 4x below the limit.
This might require tuning the maximum number of orphan sockets. What value is reasonable depends on the situation at hand. You could observe the number of orphan sockets using '/proc/net/sockstat' during peak traffic, multiply by four to get an idea of how many sockets are required.
That new value could be added to '/proc/sys/net/ipv4/tcp_max_orphans' and 'net.ipv4.tcp_max_orphans' in '/etc/sysctl.conf' so that the change persists across reboots. This change may require more physical memory as well.