The purpose of this article is to help the investigation of syslog connection issues.
It doesn't cover the configuration of syslog sources or destinations.
Syslog connection in the view of Syslog-ng is a client-server connection to transfer standard BSD or IETF syslog formatted messages, using network() or syslog() driver.
Syslog-ng generates status and error messages of syslog connections on the internal() source.
Checking these messages in the system logs can help to find the root cause of a failed syslog connection.
The log message of a syslog connection includes the state, the IP:PORT information and the file descriptor (fd) of the connection.
The basic syslog connection messages are the followings.
Syslog connection accepted
The syslog 'client=IP:PORT' connected to syslog-ng source listening on 'local=IP:PORT'. IP address 0.0.0.0 means that syslog-ng is listening on PORT on all available local IP addresses.
Syslog connection accepted; fd='20', client='AF_INET(10.10.10.20:56928)', local='AF_INET(0.0.0.0:514)'
A syslog connection to a remote 'server=IP:PORT' is established. Local IP:PORT is usually 0.0.0.0:0, the exact IP and port can be checked with network tools, eg. netstat.
Syslog connection established; fd='11', server='AF_INET(10.10.10.10:514)', local='AF_INET(0.0.0.0:0)'
A syslog connection is closed normally by one of the peer, that can be either the client or the server.
Connection from 'client=IP:PORT' to 'local=IP:PORT' closed.
Syslog connection closed; fd='24', client='AF_INET(10.10.10.20:45230)', local='AF_INET(10.10.10.10:514)'
Remote connection to 'server=IP:PORT' closed.
Syslog connection closed; fd='24', server='AF_INET(10.10.10.10:514)', time_reopen='60'
Syslog connection broken
A remote syslog connection to 'server=IP:PORT' was terminated. It can be caused by various reasons, for example RLTP timeout or network issue.
Syslog connection broken; fd='15', server='AF_INET(10.10.10.10:514)', time_reopen='60'
Syslog connection failed
A remote syslog connection to 'server=IP:PORT' is failed. The error message is usually a standard Linux system error.
Syslog connection failed; fd='80', server='AF_INET(10.10.10.10:6514)', error='No route to host (113)', time_reopen='60'
Usually it happens when the remote server is not accessible because of a network outage or there may be a typo in the configuration.
In such cases syslog-ng client will drop a "Syslog connection failed" message. The connection doesn't reach the server, so there is no log message can be found.
When a protocol related issue happens, it is recommended to check the logs of both the syslog server and client. In many cases the syslog server gives more exact error messages.
In such cases the connection can be established, but the server does not accept the log because of a protocol error.
The syslog-ng server may refuse the connection, because of various reasons e.g. license or max-connections() limit has been reached. In such cases, before closing the connection the server logs an error message with the reason of rejection.
Before troubleshooting a syslog connection it is recommended to double-check the followings.
The following example shows the log messages of a connection which tries to use incorrect format. The server closes the connection because the client sends BSD syslog messages instead of IETF syslog.
Syslog client messages
May 13 12:15:07 syslog-client syslog-ng: Syslog connection established; fd='18', server='AF_INET(10.10.10.10:601)', local='AF_INET(0.0.0.0:0)'
May 13 12:15:07 syslog-client syslog-ng: EOF occurred while idle; fd='18'
May 13 12:15:07 syslog-client syslog-ng: Syslog connection broken; fd='18', server='AF_INET(10.10.10.10:601)', time_reopen='60'
Syslog server messages
2019-05-13T12:15:07+02:00 syslog-server syslog-ng: Syslog connection accepted; fd='50', client='AF_INET(10.1.1.30:46746)', local='AF_INET(10.10.10.10:601)'
2019-05-13T12:15:07+02:00 syslog-server syslog-ng: Invalid frame header; header=''
2019-05-13T12:15:07+02:00 syslog-server syslog-ng: Syslog connection closed; fd='50', client='AF_INET(10.1.1.30:46746)', local='AF_INET(10.10.10.10:601)'
Syslog-ng may produce lots of internal messages on a loaded system. Syslog connection tracing can be done with the following method.
Syslog message formats: https://support.oneidentity.com/syslog-ng-premium-edition/kb/264126
Common connection issues of syslog-ng: https://support.oneidentity.com/syslog-ng-premium-edition/kb/284050
Common issues of TLS encrypted message transfer: https://support.oneidentity.com/syslog-ng-premium-edition/kb/263658