The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to syslog-ng OSE crashes if the client and the destination server communicate using the
Normal disk queues work in a different way: they employ an in-memory output buffer (set in qout-size()) and an in-memory overflow queue (set in mem-buf-length()). The disk buffer file itself is only used if the in-memory output buffer (set in qout-size()) is filled up completely. This approach has better performance (because of less disk IO operations), but also carries the risk of losing a maximum of qout-size() plus mem-buf-length() number of messages in case of an unexpected power failure or application crash.
Disk queue files tend to grow. Each may take up to disk-buf-size() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file. Truncation only occurs if the read and write heads of the queue reach the same position. Given that new messages arrive all the time, at least a small number of messages will almost always be stored in the queue file at all times. As a result, the queue file is not truncated automatically, but grows until it reaches the maximal configured size, after which the write head will wrap around, later followed by the read head.
In case of normal disk queue files, growth in size is not so apparent, as the disk-based queue file is only used if the in-memory overflow buffer fills up. Once the destination sends messages faster than the incoming message rate, the queue will start to empty, and when the read and write heads of the queue reach the same position, the queue files are finally truncated.
Note that if a queue file becomes corrupt, syslog-ng OSE starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.
The following sections describe how to select and filter log messages.
Using filters describes how to configure and use filters.
Combining filters with boolean operators shows how to create complex filters using boolean operators.
Comparing macro values in filters explains how to evaluate macros in filters.
Using wildcards, special characters, and regular expressions in filters provides tips on using regular expressions.
Tagging messages explains how to tag messages and how to filter on the tags.
Filter functions is a detailed description of the filter functions available in syslog-ng OSE.
Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.
To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:
filter <identifier> { <filter_type>("<filter_expression>"); };
Then use the filter in a log path, for example:
log {
source(s1);
filter(<identifier>);
destination(d1); };
You can also define the filter inline. For details, see Defining configuration objects inline.
The following filter statement selects the messages that contain the word deny and come from the host example.
filter demo_filter { host("example") and match("deny" value("MESSAGE"))
};
log {
source(s1);
filter(demo_filter);
destination(d1);
};
The following example does the same, but defines the filter inline.
log {
source(s1);
filter { host("example") and match("deny" value("MESSAGE")) };
destination(d1);
};When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time):
filter demo_filter1 { host("example1"); };
filter demo_filter2 { host("example2"); };
log {
source(s1); source(s2);
filter(demo_filter1); filter(demo_filter2);
destination(d1); destination(d2); };
To select the messages that come from either host example1 or example2, use a single filter expression:
filter demo_filter { host("example1") or host("example2"); };
log {
source(s1); source(s2);
filter(demo_filter);
destination(d1); destination(d2); };
Use the not operator to invert filters, for example, to select the messages that were not sent by host example1:
filter demo_filter { not host("example1"); };
However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that's how boolean logic works):
filter demo_filter { not host("example1") and not host("example2"); };
Alternatively, you can use parentheses to avoid this confusion:
filter demo_filter { not (host("example1") or host("example2")); };
For a complete description on filter functions, see Filter functions.
The following filter statement selects the messages that contain the word deny and come from the host example.
filter demo_filter { host("example") and match("deny" value("MESSAGE")); };
The value() parameter of the match function limits the scope of the function to the text part of the message (that is, the part returned by the ${MESSAGE} macro), or optionally to the content of any other macro. The template() parameter of the match function can be used to run a filter against a combination of macros. For details on using the match() filter function, see match().
Filters are often used together with log path flags. For details, see Log path flags.
© 2024 One Identity LLC. ALL RIGHTS RESERVED. Conditions d’utilisation Confidentialité Cookie Preference Center