syslog-ng Open Source Edition 3.22 - Administration Guide

Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages.

If you use flow-control and reliable disk-based buffering together with multiple destinations, the flow-control starts slowing down the source only when:

• one destination is down, and

• the number of messages stored in the disk buffer of the destination reaches (disk-buf-size() minus mem-buf-size()).

Configuring flow-control

For details on how flow-control works, see Managing incoming and outgoing messages with flow-control. The summary of the main points is as follows:

• The syslog-ng application normally reads a maximum of log-fetch-limit() number of messages from a source.

• From TCP and unix-stream sources, syslog-ng reads a maximum of log-fetch-limit() from every connection of the source. The number of connections to the source is set using the max-connections() parameter.

• Every destination has an output buffer. The size of this buffer is set automatically for log paths that use flow-control, and can be set using the log-fifo-size() option for other log paths.

• Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window, the log-iw-size() option sets the size of the static control window. Optionally, you can enable a dynamic control window for the source using the dynamic-window-size() option.

• When a source accepts multiple connections, the size of the control window is divided by the value of the max-connections() parameter and this smaller control window is applied to each connection of the source.

  The dynamic control window is automatically distributed among the active connections of the source.

• If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.

• If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages may be lost.

source s_localhost {
    network(
        ip(127.0.0.1)
        port(1999)
        max-connections(300)
    );
};
destination d_tcp {
    network("10.1.2.3"
        port(1999)
        localport(999)
        log-fifo-size(30000)
    );
};
log {
    source(s_localhost);
    destination(d_tcp);
    flags(flow-control);
};

source s_localhost {
    network(
        ip(127.0.0.1)
        port(1999)
        max-connections(300)
    );
};
source s_tcp {
    network(
        ip(192.168.1.5)
        port(1999)
        max-connections(100)
    );
};
destination d_tcp {
    network("10.1.2.3"
        port(1999)
        localport(999)
        log-fifo-size(40000)
    );
};
log {
    source(s_localhost);
    destination(d_tcp);
    flags(flow-control);
};

Using disk-based and memory buffering

The syslog-ng Open Source Edition application can store messages on the local hard disk if the destination (for example, the central log server) or the network connection to the destination becomes unavailable. The syslog-ng OSE application automatically sends the stored messages to the destination when the connection is reestablished. The disk buffer is used as a queue: when the connection to the destination is reestablished, syslog-ng OSE sends the messages to the destination in the order they were received.

Every such destination uses a separate disk buffer (similarly to the output buffers controlled by log-fifo-size()). The hard disk space is not pre-allocated, so ensure that there is always enough free space to store the disk buffers even when the disk buffers are full.

If syslog-ng OSE is restarted (using the /etc/init.d/syslog-ng restart command, or another appropriate command on your platform), it automatically saves any unsent messages from the disk buffer and the output queue. After the restart, syslog-ng OSE sends the saved messages to the destination. In other words, the disk buffer is persistent. The disk buffer is also resistant to syslog-ng OSE crashes.

The syslog-ng OSE application supports two types of disk buffering: reliable and normal. For details, see Enabling reliable disk-based buffering and Enabling normal disk-based buffering, respectively.

Message handling and normal disk-based buffering

When you use disk-based buffering, and the reliable() option is set to no, syslog-ng OSE handles outgoing messages the following way:

Figure 16: Handling outgoing messages in syslog-ng OSE

• Output queue: Messages from the output queue are sent to the destination (for example, your central log server). The syslog-ng OSE application puts the outgoing messages directly into the output queue, unless the output queue is full. By default, the output queue can hold 64 messages (you can adjust it using the quot-size() option).

• Disk buffer: If the output queue is full, disk-buffering is enabled, and reliable() is set to no, syslog-ng OSE puts the outgoing messages into the disk buffer of the destination. (The disk buffer is enabled if the disk-buffer() option is configured.)

• Overflow queue: If the output queue is full and the disk buffer is disabled or full, syslog-ng OSE puts the outgoing messages into the overflow queue of the destination. (The overflow queue is identical to the output buffer used by other destinations.) The log-fifo-size() parameter specifies the number of messages stored in the overflow queue. For details on sizing the log-fifo-size() parameter, see also Managing incoming and outgoing messages with flow-control.

NOTE: Using disk buffer can significantly decrease performance.

Enabling reliable disk-based buffering

To enable reliable disk-based buffering, use the disk-buffer(reliable(yes)) parameter in the destination. Use reliable disk-based buffering if you do not want to lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. The filename of the reliable disk buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.rqf.

destination d_BSD {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
        )
    );
}; 

For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.