The syslog-ng OSE application can parse fields from a message with the help of regular expressions. This can be also achieved with the match() filter, by setting the store-matches flag, but the regexp-parser() offers more flexibility, like multiple patterns and setting the prefix of the created name-value pairs.
NOTE: The regexp-parser() can create additional name-value pairs only if "named capture groups" are used in the regular expression, for example (?<test_field>\w+). For more information, see "named capture groups" in PCRE documentation.
For more information about regular expressions in syslog-ng OSE, see Regular expressions.
For example:
Declaration:
parser p_regexp {
regexp-parser(
patterns( ... )
);
};
Example: Using a regexp-parser()
In the following example, the incoming log message is the following:
Apr 20 11:09:46 test_field -> test_value
The regexp-parser inserts the .regexp. prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log line connects the source, the parser and the destination.
source s_network {
network(
port(21514)
flags(no-parse)
);
};
parser p_regexp {
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$")
prefix(".regexp.")
);
};
destination d_file {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_network);
parser(p_regexp);
destination(d_file);
};
You can also define the parser inline in the log path.
source s_network {
network(
port(21514)
flags(no-parse)
);
};
destination d_file {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_network);
parser{
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$")
prefix(".regexp.")
);
};
destination(d_file);
};
You can set multiple patterns:
parser p_regexp {
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$", ".*other_format: (?<foo>.*)$")
prefix(".regexp.")
);
};
The Regular expression parser has the following options.
flags()
| Type: |
assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8 |
| Default: |
empty set |
Description: Specifies the log parsing options of the source.
-
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
-
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.
-
expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
-
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
-
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
-
no-header: The no-header flag triggers syslog-ng OSE to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
NOTE: Essentially, the no-header flag signals syslog-ng OSE that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
parser p_syslog {
syslog-parser(
flags(no-header)
);
};
-
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
source s_dell {
network(
port(2000)
flags(no-hostname)
);
};
-
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
-
no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng OSE generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
-
dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
-
sanitize-utf8: When using the sanitize-utf8 flag, syslog-ng OSE converts non-UTF-8 input to an escaped form, which is valid UTF-8.
-
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another syslog-ng node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
-
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
-
validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see IETF-syslog messages). If the BOM character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.
patterns()
| Synopsis: |
patterns("pattern1" "pattern2") |
| Mandatory: |
yes |
Description: The regular expression patterns that you want to find a match. regexp-parser() supports multiple patterns, and stops the processing at the first successful match.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser p_regexp{
regexp-parser(
patterns( ... )
prefix("myprefix.")
);
};
template()
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
The syslog-ng application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example, can label the messages as user login, application crash, file transfer, and so on events.
To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.
To classify a message, syslog-ng selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.
To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example, 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Using pattern parsers.
The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand syslog-ng pattens than regular expressions.
Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2
A regular expression matching this message from the logcheck project: Accepted (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?
A syslog-ng database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2
For details on using pattern databases to classify log messages, see Using pattern databases.
