Extensible Markup Language (XML) is a text-based open standard designed for both human-readable and machine-readable data interchange. Like JSON, it is used primarily to transmit data between a server and web application. It is described in W3C Recommendation: Extensible Markup Language (XML).
The XML parser processes input in XML format, and adds the parsed data to the message object.
To create an XML parser, define an xml_parser that has the xml() option. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message using the XML parser, use the template() option. You can also define the parser inline in the log path.
Declaration:
parser xml_name {
xml(
template()
prefix()
drop-invalid()
exclude-tags()
strip-whitespaces()
);
};
Example: Using an XML parser
In the following example, the source is an XML-encoded log message. The destination is a file that uses the format-json template. The log line connects the source, the destination and the parser.
source s_local {
file("/tmp/aaa");
};
destination d_local {
file(
"/tmp/bbb"
template("$(format-json .xml.*)\n")
);
};
parser xml_parser {
xml();
};
log {
source(s_local);
parser(xml_parser);
destination(d_local);
};
You can also define the parser inline in the log path.
log {
source(s_file);
parser { xml(prefix(".SDATA")); };
destination(d_file);
};
The XML parser inserts an ".xml" prefix by default before the extracted name-value pairs. Since format-json replaces a dot with an underscore at the beginning of keys, the ".xml" prefix becomes "_xml". Attributes get an _ prefix. For example, from the XML input:
<tags attr='attrval'>part1<tag1>Tag1 Leaf</tag1>part2<tag2>Tag2 Leaf</tag2>part3</tags>
The following output is generated:
{"_xml":{"tags":{"tag2":"Tag2 Leaf","tag1":"Tag1 Leaf","_attr":"attrval","tags":"part1part2part3"}}}
When the text is separated by tags on different levels or tags on the same level, the parser simply concatenates the different parts of text. For example, from this input XML:
<tag>
<tag1>text1</tag1>
<tag1>text2</tag1>
</tag>
The following output is generated:
.xml.tag.tag1 = text1text2
Whitespaces are kept as they are in the XML input. No collapsing happens on significant whitespaces. For example, from this input XML:
<133>Feb 25 14:09:07 webserver syslogd: <b>|Test\n\n Test2|</b>\n
The following output is generated:
[2017-09-04T13:20:27.417266] Setting value; msg='0x7f2fd8002df0', name='.xml.b', value='|Test\x0a\x0a Test2|'
However, note that users can choose to strip whitespaces using the strip-whitespaces() option.
Configuration hints
Define a source that correctly detects the end of the message, otherwise the XML parser will consider the input invalid, resulting in a parser error.
To ensure that the end of the XML document is accurately detected, use any of the following options:
In case you experience issues, start syslog-ng with debug logs enabled. There will be a debug log about the incoming log entry, which shows the complete message to be parsed. The entry should contain the entire XML document.
NOTE: If your log messages are entirely in .xml format, make sure to disable any message parsing on the source side by including the flags("no-parse") option in your source statement. This will put the entire log message in the $MESSAGE macro, which is the field that the XML parser parses by default.
The XML parser comes with certain limitations.
Vector-like structures:
It is not possible to address each element of a vector-like structure individually. For example, take this input:
<vector>
<entry>value1</entry>
<entry>value2</entry>
...
<entry>valueN</entry>
</vector>
After parsing, the entries cannot be addressed individually. Instead, the text of the entries will be concatenated:
vector.entry = "value1value2...valueN"
Note that xmllint has the same behavior:
$ xmllint --xpath "/vector/entry/text()" test.xml
value1value2valueN%
CDATA:
The XML parser does not support CDATA. CDATA inside the XML input is ignored. This is true for the processing instructions as well.
Inherited limitations:
The XML parser is based on the glib XML subset parser, called "GMarkup" parser, which is not a full-scale XML parser. It is intended to parse a simple markup format that is a subset of XML. Some limitations are inherited:
-
Do not use the XML parser if you expect to interoperate with applications generating full-scale XML. Instead, use it for application data files, configuration files, log files, and so on, where you know your application will be the only one writing the file.
-
The XML parser is not guaranteed to display an error message in the case of invalid XML. It may accept invalid XML. However, it does not accept XML input that is not well-formed (a condition that is weaker than requiring XML to be valid).
No support for long keys:
If the key is longer than 255 characters, syslog-ng drops the entry and an error log is emitted. There is no chunking or any other way of recovering data, not even partial data. The entry will be replaced by an empty string.
The XML parser has the following options.
drop-invalid
| Synopsis: |
drop-invalid() |
| Format: |
yes|no |
| Default: |
no |
| Mandatory: |
no |
Description: If set, messages with an invalid XML will be dropped entirely.
exclude-tags
| Synopsis: |
exclude-tags() |
| Format: |
list of globs |
| Default: |
None
If not set, no filtering is done. |
| Mandatory: |
no |
Description: The XML parser matches tags against the listed globs. If there is a match, the given subtree of the XML will be omitted.
Example: Using exclude_tags
parser xml_parser {
xml(
template("$MSG")
exclude-tags("tag1", "tag2", "inner*")
);
};
From this XML input:
<tag1>Text1</tag1><tag2>Text2</tag2><tag3>Text3<innertag>TextInner</innertag></tag3>
The following output is generated:
{"_xml":{"tag3":"Text3"}}
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
The prefix() option is optional and its default value is ".xml".
strip-whitespaces
| Synopsis: |
strip-whitespaces() |
| Format: |
yes|no |
| Default: |
no |
| Mandatory: |
no |
Description: Strip the whitespaces from the XML text nodes before adding them to the message.
Example: Using strip-whitespaces
parser xml_parser {
xml(
template("$MSG")
strip-whitespaces(yes)
);
};
From this XML input:
<tag1> Tag </tag1>
The following output is generated:
{"_xml":{"tag1":"Tag"}}
template()
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
The date parser can extract dates from non-syslog messages. It operates by default on the ${MESSAGE} part of the log message, but can operate on any template or field provided. The parsed date will be available as the sender date (that is, the ${S_DATE}, ${S_ISODATE}, ${S_MONTH}, and so on, and related macros). (To store the parsed date as the received date, use the time-stamp(recvd) option.)
NOTE: Note that parsing will fail if the format string does not match the entire template or field. Since by default syslog-ng Open Source Edition (syslog-ng OSE) uses the ${MESSAGE} part of the log message, parsing will fail, unless the log message contains only a date, but that is unlikely, so practically you will have to segment the message (for example, using a csv-parser()) before using the date-parser(). You can also use date-parser() to parse dates received in a JSON or key-value-formatted log message.
Declaration
parser parser_name {
date-parser(
format("<format-string-for-the-date>")
template("<field-to-parse>'")
);
};
Example: Using the date-parser()
In the following example, syslog-ng OSE parses dates like 01/Jan/2016:13:05:05 PST from a field called MY_DATE using the following format string: format("%d/%b/%Y:%H:%M:%S %Z") (how you create this field from the incoming message is not shown in the example). In the destination template every message will begin with the timestamp in ISODATE format. Since the syslog parser is disabled, syslog-ng OSE will include the entire original message (including the original timestamp) in the ${MESSAGE} macro.
source s_file {
file("/tmp/input" flags(no-parse));
};
destination d_file {
file(
"/tmp/output"
template("${S_ISODATE} ${MESSAGE}\n")
);
};
log {
source(s_file);
parser { date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("${MY_DATE}")); };
destination(d_file);
};
In the template option, you can use template functions to specify which part of the message to parse with the format string. The following example selects the first 24 characters of the ${MESSAGE} macro.
date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("$(substr ${MESSAGE} 0 24)") );
In syslog-ng OSE version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:
date-parser(format(
"%FT%T.%f",
"%F %T,%f",
"%F %T"
));
If you need to modify or correct the timezone of the message after parsing, see Rewrite the timezone of a message.
