The XML parser has the following options.
| Synopsis: | drop-invalid() |
| Format: | yes|no |
| Default: | no |
| Mandatory: | no |
Description: If set, messages with an invalid XML will be dropped entirely.
| Synopsis: | exclude-tags() |
| Format: | list of globs |
| Default: |
None If not set, no filtering is done. |
| Mandatory: | no |
Description: The XML parser matches tags against the listed globs. If there is a match, the given subtree of the XML will be omitted.
parser xml_parser {
xml(
template("$MSG")
exclude-tags("tag1", "tag2", "inner*")
);
};
From this XML input:
<tag1>Text1</tag1><tag2>Text2</tag2><tag3>Text3<innertag>TextInner</innertag></tag3>
The following output is generated:
{"_xml":{"tag3":"Text3"}}| Synopsis: | prefix() |
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard vs. soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
The prefix() option is optional and its default value is ".xml".
| Synopsis: | strip-whitespaces() |
| Format: | yes|no |
| Default: | no |
| Mandatory: | no |
Description: Strip the whitespaces from the XML text nodes before adding them to the message.
parser xml_parser {
xml(
template("$MSG")
strip-whitespaces(yes)
);
};
From this XML input:
<tag1> Tag </tag1>
The following output is generated:
{"_xml":{"tag1":"Tag"}}| Synopsis: | template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
The date parser can extract dates from non-syslog messages. It operates by default on the ${MESSAGE} part of the log message, but can operate on any template or field provided. The parsed date will be available as the sender date (that is, the ${S_DATE}, ${S_ISODATE}, ${S_MONTH}, and so on, and related macros). (To store the parsed date as the received date, use the time-stamp(recvd) option.)
Note that parsing will fail if the format string does not match the entire template or field. Since by default syslog-ng OSE uses the ${MESSAGE} part of the log message, parsing will fail, unless the log message contains only a date, but that is unlikely, so practically you will have to segment the message (for example, using a csv-parser()) before using the date-parser(). You can also use date-parser() to parse dates received in a JSON or key-value-formatted log message.
parser parser_name {
date-parser(
format("<format-string-for-the-date>")
template("<field-to-parse>'")
);
};In the following example, syslog-ng OSE parses dates like 01/Jan/2016:13:05:05 PST from a field called MY_DATE using the following format string: format("%d/%b/%Y:%H:%M:%S %Z") (how you create this field from the incoming message is not shown in the example). In the destination template every message will begin with the timestamp in ISODATE format. Since the syslog parser is disabled, syslog-ng OSE will include the entire original message (including the original timestamp) in the ${MESSAGE} macro.
source s_file {
file("/tmp/input" flags(no-parse));
};
destination d_file {
file(
"/tmp/output"
template("${S_ISODATE} ${MESSAGE}\n")
);
};
log {
source(s_file);
date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("${MY_DATE}") );
destination(d_file);
};
In the template option, you can use template functions to specify which part of the message to parse with the format string. The following example selects the first 24 characters of the ${MESSAGE} macro.
date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("$(substr ${MESSAGE} 0 24)") );The date-parser() parser has the following options.
| Synopsis: | format(string) |
| Default: |
Description: Specifies the format how syslog-ng OSE should parse the date. You can use the following format elements:
%% PERCENT %a day of the week, abbreviated %A day of the week %b month abbr %B month %c MM/DD/YY HH:MM:SS %C ctime format: Sat Nov 19 21:05:57 1994 %d numeric day of the month, with leading zeros (eg 01..31) %e like %d, but a leading zero is replaced by a space (eg 1..31) %D MM/DD/YY %G GPS week number (weeks since January 6, 1980) %h month, abbreviated %H hour, 24 hour clock, leading 0's) %I hour, 12 hour clock, leading 0's) %j day of the year %k hour %l hour, 12 hour clock %L month number, starting with 1 %m month number, starting with 01 %M minute, leading 0's %n NEWLINE %o ornate day of month -- "1st", "2nd", "25th", etc. %p AM or PM %P am or pm (Yes %p and %P are backwards :) %q Quarter number, starting with 1 %r time format: 09:05:57 PM %R time format: 21:05 %s seconds since the Epoch, UCT %S seconds, leading 0's %t TAB %T time format: 21:05:57 %U week number, Sunday as first day of week %w day of the week, numerically, Sunday == 0 %W week number, Monday as first day of week %x date format: 11/19/94 %X time format: 21:05:57 %y year (2 digits) %Y year (4 digits) %Z timezone in ascii. eg: PST %z timezone in format -/+0000
For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: format("%d/%b/%Y:%H:%M:%S %Z")
| Synopsis: | template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
| Synopsis: | stamp | recvd |
| Default: | stamp |
Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(stamp), syslog-ng OSE adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-stamp(recvd), syslog-ng OSE adds the parsed date to the R_ macros (corresponding to the received date).
| Synopsis: | time-zone(string) |
| Default: |
Description: If this option is set, syslog-ng OSE assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
The Apache Access Log Parser can parse the access log messages of the Apache HTTP Server. The syslog-ng OSE application can separate these log messages to name-value pairs. For details on using value-pairs in syslog-ng OSE see Structuring macros, metadata, and other value-pairs. The apache-accesslog-parser() supports both the Common Log Format and the Combined Log Format of Apache (for details, see the Apache HTTP Server documentation). The following is a sample log message:
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
The syslog-ng OSE application extracts every field into name-value pairs, and adds the .apache. prefix to the name of the field.
parser parser_name {
apache-accesslog-parser(
prefix()
);
};The parser extracts the following fields from the messages: clientip, ident, auth, timestamp, rawrequest, response, bytes, referrer, and agent. The rawrequest field is further segmented into the verb, request, and httpversion fields. The syslog-ng OSE apache-accesslog-parser() parser uses the same naming convention as Logstash.
In the following example, the source is a log file created by an Apache web server. The parser automatically inserts ".apache." prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log statement connects the source, the destination, and the parser.
source s_apache {
file(/var/log/access_log);
};
destination d_json {
file(
"/tmp/test.json"
template("$(format-json .apache.*)\n")
);
};
log {
source(s_apache);
parser { apache-accesslog-parser();};
destination(d_json);
};To use this parser, the scl.conf file must be included in your syslog-ng OSE configuration:
@include "scl.conf"
The apache-accesslog-parser() is actually a reusable configuration snippet configured parse Apache access log messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
© 2024 One Identity LLC. ALL RIGHTS RESERVED. 使用条款 隐私 Cookie Preference Center