-
Title
How to create alert on failed syslog connection to a remote server -
Description
This knowledge article shows how to set up content based alert to get notified when a remote server is not accessible. -
Cause
When a remote server becomes unavailable, syslog-ng starts to buffer log messages.
Syslog-ng Store Box (SSB) keeps receiving logs and store them in buffers.
If flow-control is set syslog-ng stops reading the sources to avoid message loss.
Note, that it even flow-control is set message loss may happen mostly if UDP protocol is used.If flow-control is not set, syslog-ng will not stop reading the sources and fills up the buffers, so message loss may happen.
-
Resolution
Syslog-ng tracks its connections and writes events in the local logspace.
A content-based alert (CBA) can be setup up based on that events.
Resolution
In this example the remote server uses the IP address 10.21.10.10 and accepts syslog messages on port 514.
Remote server IP and port can be found as 'Address' and 'Port' at Log | Destinations | Destination name.- Enable 'default_kv_parser' for the logpath of the 'local' logspace at Log | Paths.
Syslog-ng will generate kv-pairs from the connection events.
Otherwise, because of the indexer configuration the searching for an IP address may be impossible or will produce other results as well.- Create content based alert(s) using the following expressions. The IP and port in bold must be changed accordingly.
SSB can not connect to the remote server.
program:syslog-ng connection failed nvpair:.sdata.kv.server=*10.21.10.10:514*
The output buffer is full, messages get dropped.
Due to indexer limitations a rewrite rule is needed to create this alert, otherwise IP address of the remote server is not fully idexed.
- Navigate to the log path of the 'local' logspace.
- Create a rewrite rule 'Before message processing' with the following values.
In message part: MESSAGE
Find: afsocket_dd_qfileprogram:syslog-ng Destination queue full nvpair:.sdata.kv.persist_name=*10.21.10.10:514
More information:
Troubleshooting syslog-ng connections KB265410.
Setting up alerts on the search interface Administration Guide.