To enable normal disk-based buffering, use the disk-buffer(reliable(no)) parameter in the destination. Use normal disk-based buffering if you want a solution that is faster than the reliable disk-based buffering. In this case, disk buffering will be less reliable and it is possible to lose logs in case of syslog-ng OSE crash. The filename of the normal disk buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.
When using the disk-buffer plugin:
destination d_BSD {
network("127.0.0.1"
port(3333)
disk-buffer(
mem-buf-length(10000)
disk-buf-size(2000000)
reliable(no)
)
);
}; For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.
To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where disk-based buffering is not available. Or if you want the fastest solution, and if syslog-ng OSE crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use disk-based buffering, logs are stored only in the memory.
destination d_BSD {
network("127.0.0.1"
port(3333)
log-fifo-size(10000)
);
};The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to syslog-ng OSE crashes if the client and the destination server communicate using the
Normal disk queues work in a different way: they employ an in-memory output buffer (set in qout-size()) and an in-memory overflow queue (set in mem-buf-length()). The disk buffer file itself is only used if the in-memory output buffer (set in qout-size()) is filled up completely. This approach has better performance (because of less disk IO operations), but also carries the risk of losing a maximum of qout-size() plus mem-buf-length() number of messages in case of an unexpected power failure or application crash.
Disk queue files tend to grow. Each may take up to disk-buf-size() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file. Truncation only occurs if the read and write heads of the queue reach the same position. Given that new messages arrive all the time, at least a small number of messages will almost always be stored in the queue file at all times. As a result, the queue file is not truncated automatically, but grows until it reaches the maximal configured size, after which the write head will wrap around, later followed by the read head.
In case of normal disk queue files, growth in size is not so apparent, as the disk-based queue file is only used if the in-memory overflow buffer fills up. Once the destination sends messages faster than the incoming message rate, the queue will start to empty, and when the read and write heads of the queue reach the same position, the queue files are finally truncated.
Note that if a queue file becomes corrupt, syslog-ng OSE starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.
The following sections describe how to select and filter log messages.
Using filters describes how to configure and use filters.
Combining filters with boolean operators shows how to create complex filters using boolean operators.
Comparing macro values in filters explains how to evaluate macros in filters.
Using wildcards, special characters, and regular expressions in filters provides tips on using regular expressions.
Tagging messages explains how to tag messages and how to filter on the tags.
Filter functions is a detailed description of the filter functions available in syslog-ng OSE.
© 2024 One Identity LLC. ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center