syslog-ng Premium Edition 7.0.18 - Administration Guide

Enabling the normal disk-buffer option

The following destination drivers can use the disk-buffer option: elasticsearch2(), file(), hdfs(), kafka(), mongodb(), program(), riemann(), smtp(),sql(), unix-dgram(), and unix-stream(). The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method. (The other destinations or protocols do not provide the necessary feedback mechanisms required for the disk-buffer option.)

To enable the normal disk-buffer option, use the disk-buffer(reliable(no)) parameter in the destination. Use the normal disk-buffer option if you want a solution that is faster than the reliable disk-buffer option. In this case, the process will be less reliable and it is possible to lose logs in case of syslog-ng PE crash. The filename of the normal disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.

destination d_BSD {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-length(10000)
                disk-buf-size(2000000)
                reliable(no)
            )
        );
        }; 

For more details on the differences between the normal and the reliable disk-buffer options, see About disk queue files.

Enabling memory buffering

To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where the disk-buffer option is not available, if you want the fastest solution, and if syslog-ng PE crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use the disk-buffer option. Instead, logs are stored only in the memory.

destination d_BSD {
    network(
            "127.0.0.1"
            port(3333)
            log-fifo-size(10000)
        );
};

About disk queue files

Normal and reliable queue files

The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to syslog-ng PE crashes if the client and the destination server communicate using the Advanced Log Transfer Protocol (ALTP). Note that the Reliable Log Transfer Protocol is available only in syslog-ng Premium Edition version 6 LTS. Of course, using the reliable(yes) option introduces a significant performance penalty as well. Reliable disk queues employ an in-memory cache buffer, the content of which is also written to the disk, and which is intended to speed up the process of reading back data from the queue.

Normal disk queues work in a different way: they employ an in-memory output buffer (set in qout-size()) and an in-memory overflow queue (set in mem-buf-length()). The disk-buffer file itself is only used if the in-memory output buffer (set in qout-size()) is filled up completely. This approach has better performance (because of less disk IO operations), but also carries the risk of losing a maximum of qout-size() plus mem-buf-length() number of messages in case of an unexpected power failure or application crash.

Size and truncation of queue files

Disk queue files tend to grow. Each may take up to disk-buf-size() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file. Truncation only occurs if the read and write heads of the queue reach the same position. Given that new messages arrive all the time, at least a small number of messages will almost always be stored in the queue file at all times. As a result, the queue file is not truncated automatically, but grows until it reaches the maximal configured size, after which the write head will wrap around, later followed by the read head.

In case of normal disk queue files, growth in size is not so apparent, as the disk-based queue file is only used if the in-memory overflow buffer fills up. Once the destination sends messages faster than the incoming message rate, the queue will start to empty, and when the read and write heads of the queue reach the same position, the queue files are finally truncated.

Note that if a queue file becomes corrupt, syslog-ng PE starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.

Filters

The following sections describe how to select and filter log messages.

• Using filters describes how to configure and use filters.

• Combining filters with boolean operators shows how to create complex filters using boolean operators.

• Comparing macro values in filters explains how to evaluate macros in filters.

• Using wildcards, special characters, and regular expressions in filters provides tips on using regular expressions.

• Tagging messages explains how to tag messages and how to filter on the tags.

• Filter functions is a detailed description of the filter functions available in syslog-ng PE.