syslog-ng Premium Edition 7.0.30 - Administration Guide

Example: Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket.

Set the max-connections() parameter of the source to 300. However, the log-fetch-limit() (default value: 10) parameter applies to every connection of the source individually, while the log-iw-size() (default value: 1000) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log-fetch-limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max-connections()*log-fetch-limit() messages to be able to read every incoming message of a poll loop. In the current example this means that log-iw-size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop.

The output buffer of the destination must accommodate at least log-iw-size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination.

source s_localhost {
            network(ip(127.0.0.1) port(1999) max-connections(300)); };
destination d_tcp {
            network("10.1.2.3" port(1999) localport(999) log-fifo-size(30000)); };
log { source(s_localhost); destination(d_tcp); flags(flow-control); };

If other sources send messages to this destination, than the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, than increase the log-fifo-size() by 10000.

source s_localhost {
            network(ip(127.0.0.1) port(1999) max-connections(300)); };
source s_tcp {
            network(ip(192.168.1.5) port(1999) max-connections(100)); };
destination d_tcp {
            network("10.1.2.3" port(1999) localport(999) log-fifo-size(40000)); };
log { source(s_localhost); destination(d_tcp); flags(flow-control); };