To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where the disk-buffer option is not available, if you want the fastest solution, and if syslog-ng PE crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use the disk-buffer option. Instead, logs are stored only in the memory.
Example: Example for using memory buffering
destination d_BSD {
network(
"127.0.0.1"
port(3333)
log-fifo-size(10000)
);
};
Normal and reliable queue files
The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to syslog-ng PE crashes if the client and the destination server communicate using the Advanced Log Transport Protocol (ALTP). Note that the Reliable Log Transfer Protocol is available only in syslog-ng Premium Edition version 6 LTS. Of course, using the reliable(yes) option introduces a significant performance penalty as well.
Both reliable and normal disk-buffers employ an in-memory output queue (set in quot-size()) and an in-memory overflow queue (set in mem-buf-size() for reliable disk-buffers, or mem-buf-length() for normal disk-buffers). The difference between reliable and normal disk-buffers is that when the reliable disk-buffer uses one of its in-memory queues, it also stores the message on the disk, whereas the normal disk-buffer stores the message only in memory. The normal disk-buffer only uses the disk if the in-memory output buffer is filled up completely. This approach has better performance (due to fewer disk I/O operations), but also carries the risk of losing a maximum of quot-size() plus mem-buf-length() number of messages in case of an unexpected power failure or application crash.
Size of the queue files
Disk queue files tend to grow. Each may take up to disk-buf-size() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file.
The disk-buffer file's size should be considered as the configured disk-buf-size() at any point of time, even if it does not have messages in it. Truncating the disk-buffer file can slow down disk I/O operations, so syslog-ng PE does not always truncate the file when it would be possible (see the truncate-size-ratio() option). If a large disk-buffer file is not desirable, you should set the disk-buf-size() option to a smaller value.
|
|
Caution:
One Identity recommends that you do not build upon the current truncating logic of the disk-buffer files, because syslog-ng PE might pre-allocate the disk-buffer files and never truncate them in the future. |
NOTE: If a queue file becomes corrupt, syslog-ng PE starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.
The following sections describe how to select and filter log messages.
Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.
To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:
filter <identifier> { <filter_type>("<filter_expression>"); };
Then use the filter in a log path, for example:
log {
source(s1);
filter(<identifier>);
destination(d1); };
You can also define the filter inline. For details, see Defining configuration objects inline.
Example: A simple filter statement
The following filter statement selects the messages that contain the word deny and come from the host example.
filter demo_filter { host("example") and match("deny" value("MESSAGE")) };
log {
source(s1);
filter(demo_filter);
destination(d1); };
The following example does the same, but defines the filter inline.
log {
source(s1);
filter { host("example") and match("deny" value("MESSAGE")) };
destination(d1); };
