How to performance tune UDP log reception of syslog-ng. UDP packets are getting dropped receiving packet receive and buffer errors. (4283052)

The purpose of this knowledge base article is to help configuring syslog-ng for high incoming UDP log traffic. Many applications and devices can only send the logs over UDP. In a large environment it can happen that syslog-ng can not handle the incoming traffic and message loss happens.

UDP packets are getting dropped.  

Use UDP only if there is no other choice.

Identify UDP message loss

Kernel's receive buffer is full

• Check the size of the kernel's receive buffer (bytes)

sysctl -a |grep rmem_max
net.core.rmem_max = 212992

• Execute the following command and check the "Recv-Q" column. It indicates the size of receive buffer for the specific socket.

netstat -unlp|grep -e PID -e syslog
Proto   Recv-Q  Send-Q    Local Address        Foreign Address    State  PID/Program name
udp      212864           0     10.21.10.20:514     0.0.0.0:*                           3503/syslog-ng  

• If the number of buffered bytes is close to the net.core.rmem_max value, the buffer is full. No more messages can be received over UDP and probably packet loss happens.

UDP packet errors

• Execute the following command from the core shell and check UDP section for errors. If the number of errors is greater than 0, UDP packet were dropped, because of full receive buffer.

netstat -su

Udp:
    6798 packets received
    29 packets to unknown port received.
    34 packet receive errors
    6833 packets sent

SOLUTIONS

Solution 1 - udp-balancer() source (recommended)

• Available from syslog-ng PE 7.0.18.
• This feature requires a recent Linux kernel, so it is supported only selected platforms.
• More information about udp-balancer() can be found in the Administration Guide.
• Example: Using the udp-balancer() driver
  Listening on 192.168.1.1 (the default port for UDP is 514):

source s_network {
    udp-balancer(
        ip("192.168.1.1")
    );
};

Solution 2 - udp-balancer() is not supported

The following solutions can be applied separately and also can be combined.

Solution 2A - Increasing kernel's receive buffer size

Add the following line or modify the existing one in /etc/sysctl.conf. Change the size based on your UDP traffic, but it's a good start to double the default size.

net.core.rmem_max = 425984

Solution 2B - Configuring multiple UDP sources

The sources can be configured listening on a unique UDP port or IP address.

Example - Different UDP port

source s_udp1 { network(transport(udp) port(514));
source s_udp2 { network(transport(udp) port(515));

Example - Different IP address on default port UDP/514
Note: Syslog-ng does not manage IP addresses. IP must be assigned in the OS.

source s_udp1 { network(ip(192.168.1.1));
source s_udp2 { network(ip(192.168.1.2));