-
Title
Performance tuning of UDP log reception on SSB -
Description
The purpose of this knowledge base article is to help to configure SSB for high incoming UDP log traffic. -
Cause
Many applications and devices can only send their logs over UDP.
In a big company environment it can happen that SSB cannot handle the incoming traffic with the default configuration so message loss happens.
-
Resolution
Before start, it is strongly recommended to check the sender hosts and devices if possible to change the sending protocol to TCP instead of UDP.
Use UDP only if you have no other choice.
Backgrounds
The SSB source called 'Legacy' accepts UDP logs on port 514 and has 60MiB of receive buffer configured by kernel parameter.
UDP connections cannot be multi-threaded, so only one CPU core can be used to accept UDP logs.
Identify UDP message loss
Kernel's receive buffer is full
- Log in to the core shell
- Execute the following command and check the "Recv-Q" column. It indicates the size of receive buffer for the specific socket.
netstat -unlp|grep -e PID -e syslog
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 62914434 0 10.21.10.20:514 0.0.0.0:* 3503/syslog-ng- If the number of buffered bytes is close to 62914560 bytes (60MiB), the buffer is full. No more messages can be received over UDP and probably packet loss happens.
UDP packet errors
- Execute the following command from the core shell and check UDP section for errors. If the number of errors is greater than 0, UDP packet were dropped, because of full receive buffer.
netstat -su
Udp:
6798 packets received
29 packets to unknown port received.
34 packet receive errors
6833 packets sentSolution 1 - Add more UDP sources
Such way SSB will be able to use different CPU cores for the different UDP sources and each UDP sources will have 60MiB of receive buffer.
- Navigate to Log --> Sources menu.
- Create a new source, select UDP as transport and add the port number. (The default "legacy" source listens on port 514.)
- The "Incoming log protocol and message format" is most probably Legacy (BSD-syslog, RFC 3164).
- Add the new source to the same path where other UDP sources are or create a new log path for it.
- Reconfigure a part of the clients to send their logs to the new source.
Solution 2 - Place a syslog-ng Premium Edition relay before the SSB
Placing a syslog-ng relay before the SSB implies the reconfiguration of the logging environment. Generally there are two options.
- Reconfigure clients to send their logs to the relay.
- Configure the SSB's IP address on the relay and give a new IP for the SSB.
Note. This solution does not contain the installation of syslog-ng PE. For that check the administration guide.
Steps- Configure the syslog-ng PE using the instructions of this knowledge article.
- Configure a syslog formatted destination in the relay pointing to the SSB
How to configure BSD-syslog and IETF-syslog message formats in syslog ng Premium Edition
How to configure BSD-syslog and IETF-syslog message formats in syslog ng Store Box - It is recommended to add a disk-buffer to the destination.
Documentation: disk-buffer() destination driver option