-
Title
Logs Are Queuing/Dropping From Syslog-ng Relay -
Description
Logs are queuing and/or dropping between the relay and the destination.
When querying syslog-ng-ctl the stats may seem similar to the following:
processed='source(s_tcp)=56727085',
processed='destination(d_tcp)=24047486',
queued='dst.network(d_tcp#0,tcp,10.10.10.10:514)=5000000',
written='dst.network(d_tcp#0,tcp,10.10.10.10:514)=164041',
dropped='dst.network(d_tcp#0,tcp,10.10.10.10:514)=18884993', -
Cause
The destination cannot accept the volume of logs being sent to it by the Syslog-ng relay. -
Resolution
Resolution 1: When using a SIEM
It is common that some SIEM products such as QRadar and Splunk have limits on the number of events per second (EPS) they can receive. When sending too many EPS the logs can start to queue and/or drop on the relay as the relay is not able to communicate the logs to the SIEM fast enough. Please work with the SIEM vendor to make changes that result in being able to ingest a higher EPS.Resolution 2: When sending to a Syslog-ng Store Box (SSB)
It is possible that the SSB is being overloaded and cannot ingest logs quickly enough. Please contact One Identity Support for further assistance in this scenario for help with troubleshooting the SSB.Resolution 3: Network issues
It is possible that the network where the Syslog-ng relay is hosted cannot send the volume of logs that the relay is trying to send. Please consult with local network teams to ensure the network can handle the traffic being sent and that the bandwidth is large enough.Resolution 4: Destination Not Available
It is possible that the destination in which the relay is sending to is not available. Please ensure the destination is online and accepting logs. Common reasons a destination would not accept logs are due to firewalls, iptables, SELinux, and services not being started. For example, if the destination is a Syslog-ng PE server, please check to ensure that Syslog-ng is running and that the correct ports are being listened on. If local logging is enabled on the Syslog-ng relay server querying the local log store should produce evidence that Syslog-ng cannot reach the destination. Please contact One Identity Support for further assistance if necessary.