-
Title
Troubleshooting syslog connections in syslog-ng Premium Edition -
Description
The purpose of this article is to help the investigation of syslog connection issues.
It doesn't cover the configuration of syslog sources or destinations.
-
Cause
-
Resolution
What is a syslog connection?
Syslog connection in the view of Syslog-ng is a client-server connection to transfer standard BSD or IETF syslog formatted messages, using network() or syslog() driver.
Syslog-ng generates status and error messages of syslog connections on the internal() source.
Checking these messages in the system logs can help to find the root cause of a failed syslog connection.
Syslog connection messages
The log message of a syslog connection includes the state, the IP:PORT information and the file descriptor (fd) of the connection.
The basic syslog connection messages are the followings.
Syslog connection accepted
The syslog 'client=IP:PORT' connected to syslog-ng source listening on 'local=IP:PORT'. IP address 0.0.0.0 means that syslog-ng is listening on PORT on all available local IP addresses.
Syslog connection accepted; fd='20', client='AF_INET(10.10.10.20:56928)', local='AF_INET(0.0.0.0:514)'
Syslog connection established
A syslog connection to a remote 'server=IP:PORT' is established. Local IP:PORT is usually 0.0.0.0:0, the exact IP and port can be checked with network tools, eg. netstat.
Syslog connection established; fd='11', server='AF_INET(10.10.10.10:514)', local='AF_INET(0.0.0.0:0)'
Syslog connection closed
A syslog connection is closed normally by one of the peer, that can be either the client or the server.
Connection from 'client=IP:PORT' to 'local=IP:PORT' closed.
Syslog connection closed; fd='24', client='AF_INET(10.10.10.20:45230)', local='AF_INET(10.10.10.10:514)'
Remote connection to 'server=IP:PORT' closed.
Syslog connection closed; fd='24', server='AF_INET(10.10.10.10:514)', time_reopen='60'
Syslog connection broken
A remote syslog connection to 'server=IP:PORT' was terminated. It can be caused by various reasons, for example RLTP timeout or network issue.
Syslog connection broken; fd='15', server='AF_INET(10.10.10.10:514)', time_reopen='60'
Syslog connection failed
A remote syslog connection to 'server=IP:PORT' is failed. The error message is usually a standard Linux system error.
Syslog connection failed; fd='80', server='AF_INET(10.10.10.10:6514)', error='No route to host (113)', time_reopen='60'
Possible root causes of failed connections
Connection issues
Usually it happens when the remote server is not accessible because of a network outage or there may be a typo in the configuration.
In such cases syslog-ng client will drop a "Syslog connection failed" message. The connection doesn't reach the server, so there is no log message can be found.
Protocol issues
When a protocol related issue happens, it is recommended to check the logs of both the syslog server and client. In many cases the syslog server gives more exact error messages.
In such cases the connection can be established, but the server does not accept the log because of a protocol error.
Server issues
The syslog-ng server may refuse the connection, because of various reasons e.g. license or max-connections() limit has been reached. In such cases, before closing the connection the server logs an error message with the reason of rejection.
Tracking a connection
Before troubleshooting a syslog connection it is recommended to double-check the followings.
- The IP, PORT and protocol configuration of the syslog client and syslog server.
- The accessibility of the syslog server from the syslog client. Use ping and telnet to check the IP and PORT of the syslog server.
The following example shows the log messages of a connection which tries to use incorrect format. The server closes the connection because the client sends BSD syslog messages instead of IETF syslog.
Syslog client messages
May 13 12:15:07 syslog-client syslog-ng[18248]: Syslog connection established; fd='18', server='AF_INET(10.10.10.10:601)', local='AF_INET(0.0.0.0:0)'
May 13 12:15:07 syslog-client syslog-ng[18248]: EOF occurred while idle; fd='18'
May 13 12:15:07 syslog-client syslog-ng[18248]: Syslog connection broken; fd='18', server='AF_INET(10.10.10.10:601)', time_reopen='60'Syslog server messages
2019-05-13T12:15:07+02:00 syslog-server syslog-ng[26321]: Syslog connection accepted; fd='50', client='AF_INET(10.1.1.30:46746)', local='AF_INET(10.10.10.10:601)'
2019-05-13T12:15:07+02:00 syslog-server syslog-ng[26321]: Invalid frame header; header=''
2019-05-13T12:15:07+02:00 syslog-server syslog-ng[26321]: Syslog connection closed; fd='50', client='AF_INET(10.1.1.30:46746)', local='AF_INET(10.10.10.10:601)'Syslog-ng may produce lots of internal messages on a loaded system. Syslog connection tracing can be done with the following method.
- Find out the file descriptor (fd) of the connection from the initial message of the connection.
Client: Syslog connection established; fd='18'
Server: Syslog connection accepted; fd='50'
- Search the messages of the connection termination with the same fd.
Client: Syslog connection broken; fd='18'
Server: Syslog connection closed; fd='50'
- As the example shows, many cases the error message does not include "Syslog connection" string or the fd. (Invalid frame header; header='').
The reason for closing the connection is usually can be found in the syslog message just before the "Syslog connection closed" message.
-
Additional Information
Syslog message formats: https://support.oneidentity.com/syslog-ng-premium-edition/kb/264126
Common connection issues of syslog-ng: https://support.oneidentity.com/syslog-ng-premium-edition/kb/284050
Common issues of TLS encrypted message transfer: https://support.oneidentity.com/syslog-ng-premium-edition/kb/263658