syslog-ng Premium Edition 7.0.24 - Administration Guide

Enabling the reliable disk-buffer option

The following destination drivers can use the disk-buffer option: elasticsearch2(), file(), hdfs(), kafka(), mongodb(), program(), riemann(), smtp(),sql(), unix-dgram(), and unix-stream(). The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method. (The other destinations or protocols do not provide the necessary feedback mechanisms required for the disk-buffer option.)

To enable using the reliable disk-buffer option, use the disk-buffer(reliable(yes)) parameter in the destination. Use the reliable disk-buffer option if you do not want to lose logs in case of reload/restart, an unreachable destination or syslog-ng PE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. The filename of the reliable disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.rqf.

destination d_BSD {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-size(10000)
                disk-buf-size(2000000)
                reliable(yes)
            )
        );
}; 

For more details on the differences between the normal and the reliable disk-buffer options, see About disk queue files.

Enabling the normal disk-buffer option

The following destination drivers can use the disk-buffer option: elasticsearch2(), file(), hdfs(), kafka(), mongodb(), program(), riemann(), smtp(),sql(), unix-dgram(), and unix-stream(). The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method. (The other destinations or protocols do not provide the necessary feedback mechanisms required for the disk-buffer option.)

To enable the normal disk-buffer option, use the disk-buffer(reliable(no)) parameter in the destination. Use the normal disk-buffer option if you want a solution that is faster than the reliable disk-buffer option. In this case, the process will be less reliable and it is possible to lose logs in case of syslog-ng PE crash. The filename of the normal disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.

destination d_BSD {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-length(10000)
                disk-buf-size(2000000)
                reliable(no)
            )
        );
        }; 

For more details on the differences between the normal and the reliable disk-buffer options, see About disk queue files.

How to get information about disk-buffer files

This section describes how to get information about disk-buffer files used in syslog-ng Premium Edition (syslog-ng PE).

Topics:

Information about disk-buffer files

This section describes information about disk-buffer files used in syslog-ng Premium Edition (syslog-ng PE).

The following list contains information about how disk-buffer files are used in syslog-ng PE:

• You can configure disk-buffer() for a remote destination in the destination() statement.

  For more information about an example of configuring disk-buffer() for a remote destination in the destination() statement, see disk-buffer().

• By default, syslog-ng PE creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().
• The filenames are generated automatically by syslog-ng PE with the extensions .qf for a normal disk-buffer and .rqf for a reliable disk-buffer.
• The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn't store information about the destination.