RLTP / ALTP connection gets broken with the following errors 'Syslog connection broken' and 'The server is unaccessible, trying again the host; host=' (4282972)

An RLTP / ALTP connection to a remote server gets broken frequently. 

Error messages - like in the example - can be found in the system logs.

Mar 5 09:58:48 testhost syslog-ng[7274]: EOF occurred while idle; fd='14'
Mar 5 09:58:48 testhost syslog-ng[7274]: Syslog connection broken; fd='14', server='AF_INET(10.10.0.1:6514)', time_reopen='60'
Mar 5 09:59:48 testhost syslog-ng[7274]: Syslog try to reconnect; fd='14', server='AF_INET(10.10.0.1:6514)', time_reopen='60'
Mar 5 09:59:48 testhost syslog-ng[7274]: The server is unaccessible, trying again the host; host='10.10.0.1'
Mar 5 09:59:48 testhost syslog-ng[7274]: Syslog connection established; fd='14', server='AF_INET(10.10.0.1:6514)', local='AF_INET(0.0.0.0:0)'

RLTP / ALTP response timeout has taken in place.

If there is no RLTP / ALTP traffic for the time specified by the RLTP / ALTP option response-timeout() the server closes the connection.

In most cases these error messages are normal on a client with few logs.

Further investigation is needed if,

• the client cannot send any log message to the server via RLTP / ALTP.
• the connection gets broken immediately after it is established.

Avoiding broken connections

Solution 1 - Increasing response timeout

The default value of response-timeout() is 60 seconds. It has to be increased on the server and on the client as well. It is not recommended to set more than 15 minutes.
Example: setting two minutes in syslog-ng PE 6.

Note: In syslog-ng PE 7 the transport driver is called altp().

Client

destination d_network_rltp {
        network(
            "10.0.0.1"
            port("5555")
            transport(
                rltp(
                   response-timeout(120)
                )
        );
};

Server

source s_network_rltp {
        network(
            ip("10.0.0.1")
            port("5555")
            transport(
                rltp (
                    response-timeout(120)
               )
        );
};

On syslog-ng Store Box the response-timeout() option cannot be modified, it is 60 seconds. If the server is Syslog-ng Store Box apply "Solution 2".

Solution 2 - Sending mark messages

The connection can be kept alive by sending mark messages from the client. Set mark-freq() in the destination that is smaller than the response-timeout().
Example: setting 50 seconds in syslog-ng PE 6

Note: In syslog-ng PE 7 the transport driver is called altp().

Client

destination d_network_rltp {
        network(
            "10.0.0.1"
            port("5555")
            transport(rltp)
            mark-freq(50)
        );
};

Note: this solution will produce the similar log lines at the destination.

Mar 27 07:56:02 testhost [meta sequenceId="92"] -- MARK --

Reliable Log Transfer Protocol™ (RLTP™) is a proprietary transport protocol that prevents message loss during connection breaks. More information about RLTP can be found here.

Advanced Log Transfer Protocol (ALTP) is the newer version of RLTP introduced in syslog-ng PE 7. More information about ALTP can be found here.