The following sections describe how to customize the names of logfiles, and also how to use templates, macros, and template functions.
The syslog-ng PE application allows you to define message templates, and reference them from every object that can use a template. Templates can include strings, macros (for example date, the hostname, and so on), and template functions. For example, you can use templates to create standard message formats or filenames. For a list of macros available in syslog-ng Premium Edition, see the section called “Macros of syslog-ng PE”. For the macros of the syslog-ng Agent for Windows application, see Administration Guide for syslog-ng Agent for Windows. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.
Declaration:
template <template-name> {
template("<template-expression>") <template-escape(yes)>;
};
Template objects have a single option called template-escape()
, which is disabled by default (template-escape(no)
). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template-escape(yes)
) causes syslog-ng to escape the '
, "
, and backslash characters from the messages.
|
NOTE:
In versions 2.1 and earlier, the template-escape() option was enabled by default. |
Macros can be included by prefixing the macro name with a $
sign, just like in Bourne compatible shells. Although using braces around macro names is not mandatory, and the "$MSG"
and "${MSG}"
formats are equivalent, using the "${MSG}"
format is recommended for clarity.
To use a literal $
character in a template, you have to escape it. In syslog-ng PE versions 4.0-4.2, use a backslash (\$
). In version 5.0 and later, use $$
.
|
NOTE:
To use a literal @ character in a template, use @@ . |
Default values for macros can also be specified by appending the :-
characters and the default value of the macro. If a message does not contain the field referred to by the macro, or it is empty, the default value will be used when expanding the macro. For example, if a message does not contain a hostname, the following macro can specify a default hostname.
${HOST:-default_hostname}
|
NOTE:
For the macros of the syslog-ng Agent for Windows application, see Administration Guide for syslog-ng Agent for Windows. |
By default, syslog-ng sends messages using the following template: ${ISODATE} ${HOST} ${MSGHDR}${MSG}\n
. (The ${MSGHDR}${MSG}
part is written together because the ${MSGHDR}
macro includes a trailing whitespace.)
|
NOTE:
Earlier versions of syslog-ng used templates and scripts to send log messages into SQL databases. Starting from version 2.1, syslog-ng natively supports direct database access using the sql() destination. For details, see the section called “sql() destination options”. |
Example 14.1. Using templates and macros
The following template (t_demo_filetemplate
) adds the date of the message and the name of the host sending the message to the beginning of the message text. The template is then used in a file destination: messages sent to this destination (d_file
) will use the message format defined in the template.
template t_demo_filetemplate {
template("${ISODATE} ${HOST} ${MSG}\n"); template-escape(no); };
destination d_file {
file("/var/log/messages" template(t_demo_filetemplate)); };
Templates can also be used inline, if they are used only at a single location. The following destination is equivalent with the previous example:
destination d_file {
file ("/var/log/messages"
template("${ISODATE} ${HOST} ${MSG}\n") template-escape(no) );
};
The following file destination uses macros to daily create separate logfiles for every client host.
destination d_file {
file("/var/log/${YEAR}.${MONTH}.${DAY}/${HOST}.log");
};
|
NOTE:
Macros can be used to format messages, and also in the name of destination files or database tables. However, they cannot be used in sources as wildcards, for example, to read messages from files or directories that include a date in their name. |
Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. Hard macros are read-only. Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. In contrast to hard macros, soft macros are writable and can be modified within syslog-ng PE, for example, using rewrite rules.
Hard and soft macros are rather similar and often treated as equivalent. Macros are most commonly used in filters and templates, which does not modify the value of the macro, so both soft and hard macros can be used. However, it is not possible to change the values of hard macros in rewrite rules or via any other means.
The following macros in syslog-ng PE are hard macros and cannot be modified: BSDTAG
, CONTEXT_ID
, DATE
, DAY
, FACILITY_NUM
, FACILITY
, FULLDATE
, HOUR
, ISODATE
, LEVEL_NUM
, LEVEL
, MIN
, MONTH_ABBREV
, MONTH_NAME
, MONTH
, MONTH_WEEK
, PRIORITY
, PRI
, RCPTID
, SDATA
, SEC
, SEQNUM
, SOURCEIP
, STAMP
, TAG
, TAGS
, TZOFFSET
, TZ
, UNIXTIME
, WEEK_DAY_ABBREV
, WEEK_DAY_NAME
, WEEK_DAY
, WEEK
, YEAR_DAY
, YEAR
.
The following macros can be modified:FULLHOST_FROM
, FULLHOST
, HOST_FROM
, HOST
, LEGACY_MSGHDR
, MESSAGE
, MSG
,MSGID
, MSGONLY
, PID
, PROGRAM
, SOURCE
. Custom values created using rewrite rules or parsers can be modified as well, just like stored matches of regular expressions ($0 ... $255).
The following macros are available in syslog-ng PE.
|
Caution:
These macros are available when syslog-ng PE successfully parses the incoming message as a syslog message, or you use some other parsing method and map the parsed values to these macros.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line) . |
Description: Typically used together with the ${HOUR12}
macro, ${AMPM}
returns the period of the day: AM for hours before mid day and PM for hours after mid day. In reference to a 24-hour clock format, AM is between 00:00-12:00 and PM is between 12:00-24:00. 12AM is midnight. Available in syslog-ng PE 3.2 and later.
BSDDATE, R_BSDDATE, S_BSDDATE
Description: Date of the message in BSD timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example Jun 13 15:58:00
. If possible, it is recommended to use ISODATE
for timestamping.
Description: Facility/priority information in the format used by the FreeBSD syslogd: a priority number followed by a letter that indicates the facility. The priority number can range from 0
to 7
. The facility letter can range from A
to Y
, where A
corresponds to facility number zero (LOG_KERN), B
corresponds to facility 1 (LOG_USER), and so on.
Description: Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example: Jun 13 15:58:00
.
Description: The day the message was sent.
Description: The name of the facility (for example, kern
) that sent the message.
Description: The numerical code of the facility (for example, 0
) that sent the message.
Description: The facility that sent the message.
Description: Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4- Warning, or 3 - Error).
Description: The content of the message.
Description: The content of the message. This is an alias of the FILE_MESSAGE
macro.
Description: Name of the log file (including its path) from where the syslog-ng PE received the message.
FULLDATE, R_FULLDATE, S_FULLDATE
Description: A nonstandard format for the date of the message using the same format as ${DATE}
, but including the year as well, for example: 2006 Jun 13 15:58:00
.
Description: The name of the source host where the message originates from.
-
If the message traverses several hosts and the chain-hostnames()
option is on, the first host in the chain is used.
-
If the keep-hostname()
option is disabled (keep-hostname(no)
), the value of the $FULLHOST macro will be the DNS hostname of the host that sent the message to syslog-ng PE (that is, the DNS hostname of the last hop). In this case the $FULLHOST and $FULLHOST_FROM macros will have the same value.
-
If the keep-hostname()
option is enabled (keep-hostname(yes)
), the value of the $FULLHOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.
|
NOTE:
The use-dns() , use-fqdn() , normalize-hostnames() , and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes) ) and the message contains a hostname. |
For details on using name resolution in syslog-ng PE, see the section called “Using name resolution in syslog-ng”.
Description: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain.
The syslog-ng PE application uses the following procedure to determine the value of the $FULLHOST_FROM
macro:
-
The syslog-ng PE application takes the IP address of the host sending the message.
-
If the use-dns()
option is enabled, syslog-ng PE attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $FULLHOST_FROM
macro. This value will be the FQDN of the host if the use-fqdn()
option is enabled, but only the hostname if use-fqdn()
is disabled.
-
If the use-dns()
option is disabled, or the address resolution fails, the ${FULLHOST_FROM}
macro will return the IP address of the sender host.
For details on using name resolution in syslog-ng PE, see the section called “Using name resolution in syslog-ng”.
Description: The hour of day the message was sent.
HOUR12, R_HOUR12, S_HOUR12
Description: The hour of day the message was sent in 12-hour clock format. See also the ${AMPM}
macro. 12AM is midnight. Available in syslog-ng PE 3.2 and later.
Description: The name of the source host where the message originates from.
-
If the message traverses several hosts and the chain-hostnames()
option is on, the first host in the chain is used.
-
If the keep-hostname()
option is disabled (keep-hostname(no)
), the value of the $HOST macro will be the DNS hostname of the host that sent the message to syslog-ng PE (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value.
-
If the keep-hostname()
option is enabled (keep-hostname(yes)
), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.
|
NOTE:
The use-dns() , use-fqdn() , normalize-hostnames() , and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes) ) and the message contains a hostname. |
For details on using name resolution in syslog-ng PE, see the section called “Using name resolution in syslog-ng”.
Description: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain.
The syslog-ng PE application uses the following procedure to determine the value of the $HOST_FROM
macro:
-
The syslog-ng PE application takes the IP address of the host sending the message.
-
If the use-dns()
option is enabled, syslog-ng PE attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $HOST_FROM
macro. This value will be the FQDN of the host if the use-fqdn()
option is enabled, but only the hostname if use-fqdn()
is disabled.
-
If the use-dns()
option is disabled, or the address resolution fails, the ${HOST_FROM}
macro will return the IP address of the sender host.
For details on using name resolution in syslog-ng PE, see the section called “Using name resolution in syslog-ng”.
ISODATE, R_ISODATE, S_ISODATE
Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00
. If possible, it is recommended to use ${ISODATE}
for timestamping. Note that syslog-ng can produce fractions of a second (for example milliseconds) in the timestamp by using the frac-digits()
global or per-destination option.
Description: The priority (also called severity) of the message, represented as a numeric value, for example, 3
. For the textual representation of this value, use the ${LEVEL}
macro. See the section called “PRIORITY or LEVEL” for details.
Description: The minute the message was sent.
Description: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.
MONTH_ABBREV, R_MONTH_ABBREV, S_MONTH_ABBREV
Description: The English abbreviation of the month name (3 letters).
MONTH_NAME, R_MONTH_NAME, S_MONTH_NAME
Description: The English name of the month name.
MONTH_WEEK, R_MONTH_WEEK, S_MONTH_WEEK
Description: The number of the week in the given month (0-5). The week with numerical value 1 is the first week containing a Monday. The days of month before the first Monday are considered week 0. For example, if a 31-day month begins on a Sunday, then the 1st of the month is week 0, and the end of the month (the 30th and 31st) is week 5.
MONTHNAME, R_MONTHNAME, S_MONTHNAME
Description: The English name of the month the message was sent, abbreviated to three characters (for example Jan, Feb, and so on).
Description: The millisecond the message was sent.
Available in syslog-ng PE version 4 F2 and later.
Description: Text contents of the log message without the program name and pid. Note that this has changed in syslog-ng version 3.0: in earlier versions this macro included the program name and the pid. In syslog-ng 3.0, the ${MSG}
macro became equivalent with the ${MSGONLY}
macro. The program name and the pid together are available in the ${MSGHDR}
macro.
If you are using the flags(no-parse)
option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse)
disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line)
.
Description: The name and the PID of the program that sent the log message in PROGRAM[PID]:
format. Includes a trailing whitespace. Note that the macro returns an empty value if both the PROGRAM and PID fields of the message are empty.
Description: A string specifying the type of the message in IETF-syslog (RFC5424-formatted) messages. For example, a firewall might use the ${MSGID} "TCPIN" for incoming TCP traffic and the ${MSGID} "TCPOUT" for outgoing TCP traffic. By default, syslog-ng PE does not specify this value, but uses a dash (-) character instead. If an incoming message includes the ${MSGID} value, it is retained and relayed without modification.
Description: Message contents without the program name or pid.
Description: The time elapsed since the computer running syslog-ng PE has booted. The value of this macro is an integer containing the time in 1/100th of the second. Note that currently syslog-ng PE can access this data only on Linux platforms. On other platforms the macro contains the time elapsed since syslog-ng PE was started.
Note that syslog-ng PE evaluates the macro every time it is processed, so even if you use the same macro for the same message, its value can be different. For example, if you use it in a filter and in a destination filename, their values will be different even for the same message.
Available in syslog-ng PE version 6.0.5 and later.
Description: The PID of the program sending the message.
Description: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslog messages.
Description: The priority (also called severity) of the message, for example, error
. For the textual representation of this value, use the ${LEVEL}
macro. See the section called “PRIORITY or LEVEL” for details.
Description: PID of the application the message came from.
Description: The name of the program sending the message. Note that the content of the ${PROGRAM} variable may not be completely trusted as it is provided by the client program that constructed the message.
Description: This is disabled by default due to performance issues. To enable it, add the next option to the global options: use-uniqid(yes);
. For details, see also the section called “use-uniqid()”. A unique ID for messages generated at reception time on the receiving host. It facilitates defining relationships between messages that are potentially distributed to different files on the same host, or different hosts.
Example 14.2. Using ${RCPTID} macro
Using the following template statement in the configuration: template demo_template { template("${DATE} ${HOST} ${PROG}: ${MSG} ID:${RCPTID}\n"); };
the outgoing message will be the following: <133>Feb 25 14:09:07 webserver syslogd: restart. ID:1
.
SDATA, .SDATA.SDID.SDNAME, STRUCTURED_DATA
Description: The syslog-ng application automatically parses the STRUCTURED-DATA part of IETF-syslog messages, which can be referenced in macros. The ${SDATA}
macro references the entire STRUCTURED-DATA part of the message, while structured data elements can be referenced using the ${.SDATA.SDID.SDNAME}
macro.Available only in syslog-ng Premium Edition 4.0 and later.
|
NOTE:
When using STRUCTURED-DATA macros, consider the following:
-
When referencing an element of the structured data, the macro must begin with the dot (.) character. For example, ${.SDATA.timeQuality.isSynced} .
-
The SDID and SDNAME parts of the macro names are case sensitive: ${.SDATA.timeQuality.isSynced} is not the same as ${.SDATA.TIMEQUALITY.ISSYNCED} . |
Example 14.3. Using SDATA macros
For example, if a log message contains the following structured data: [exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]
you can use macros like: ${.SDATA.exampleSDID@0.eventSource}
— this would return the Application
string in this case.
Description: The second the message was sent.
Description: The ${SEQNUM}
macro contains a sequence number for the log message. The value of the macro depends on the scenario, and can be one of the following:
-
If syslog-ng PE receives a message via the IETF-syslog protocol that includes a sequence ID, this ID is automatically available in the ${SEQNUM}
macro.
-
If the message is a Cisco IOS log message using the extended timestamp format, then syslog-ng PE stores the sequence number from the message in this macro. If you forward this message the IETF-syslog protocol, syslog-ng PE includes the sequence number received from the Cisco device in the ${.SDATA.meta.sequenceId}
part of the message.
|
NOTE:
To enable sequence numbering of log messages on Cisco devices, use the following command on the device (available in IOS 10.0 and later): service sequence-numbers. For details, see the manual of your Cisco device. |
-
For locally generated messages (that is, for messages that are received from a local source, and not from the network), syslog-ng PE calculates a sequence number when sending the message to a destination (it is not calculated for relayed messages).
-
The sequence number is not global, but per-destination. Essentially, it counts the number of messages sent to the destination.
-
This sequence number increases by one for every message sent to the destination, and is not lost even when syslog-ng PE is reloaded or restarted.
-
This sequence number is added to every message that uses the IETF-syslog protocol (${.SDATA.meta.sequenceId}
), and can be added to BSD-syslog messages using the ${SEQNUM}
macro.
|
NOTE:
If you need a sequence number for every log message that syslog-ng PE receives, use the RCPTID macro. |
Description: The identifier of the source statement in the syslog-ng PE configuration file that received the message. For example, if syslog-ng PE received the log message from the source s_local { internal(); };
source statement, the value of the ${SOURCE} macro is s_local
. This macro is mainly useful for debugging and troubleshooting purposes.
Description: IP address of the host that sent the message to syslog-ng. (That is, the IP address of the host in the ${FULLHOST_FROM}
macro.) Please note that when a message traverses several relays, this macro contains the IP of the last relay.
Description: A timestamp formatted according to the ts-format()
global or per-destination option.
Description: The time elapsed since the syslog-ng PE instance was started (that is, the uptime of the syslog-ng PE process). The value of this macro is an integer containing the time in 1/100th of the second. For the uptime of the host running syslog-ng PE see the section called “OSUPTIME”.
Note that syslog-ng PE evaluates the macro every time it is processed, so even if you use the same macro for the same message, its value can be different. For example, if you use it in a filter and in a destination filename, their values will be different even for the same message.
Available in syslog-ng PE version 4 F1 and later.
Description: The priority and facility encoded as a 2 digit hexadecimal number.
Description: A comma-separated list of the tags assigned to the message. Available only in syslog-ng Premium Edition 3.2 and later.
|
NOTE:
Note that the tags are not part of the log message and are not automatically transferred from a client to the server. For example, if a client uses a pattern database to tag the messages, the tags are not transferred to the server. A way of transferring the tags is to explicitly add them to the log messages using a template and the ${TAGS} macro, or to add them to the structured metadata part of messages when using the IETF-syslog message format.
When sent as structured metadata, it is possible to reference to the list of tags on the central server, and for example, to add them to a database column. |
TZOFFSET, R_TZOFFSET, S_TZOFFSET
Description: The time-zone as hour offset from GMT, for example: -07:00
. In syslog-ng 1.6.x this used to be -0700
but as ${ISODATE}
requires the colon it was added to ${TZOFFSET}
as well.
UNIXTIME, R_UNIXTIME, S_UNIXTIME
Description: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00
.
Description: The microsecond the message was sent.
Available in syslog-ng PE version 4 F2 and later.
Description: The year the message was sent.
Description: The week number of the year, prefixed with a zero for the first nine week of the year. (The first Monday in the year marks the first week.)
WEEK_ABBREV, R_WEEK_ABBREV, S_WEEK_ABBREV
Description: The 3-letter English abbreviation of the name of the day the message was sent, for example Thu
.
WEEK_DAY, R_WEEK_DAY, S_WEEK_DAY
Description: The day of the week as a numerical value (1-7).
WEEK_DAY_NAME, R_WEEK_DAY_NAME, S_WEEK_DAY_NAME
Description: The English name of the day.
Template functions of syslog-ng PE
The following template functions are available in syslog-ng PE.
Syntax:
$(echo argument)
Description: Returns the value of its argument. Using $(echo ${HOST})
is equivalent to ${HOST}
.
syslog-ng PE version 5 F6 includes a new template function (format-cef-extension
) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.
You can use the value-pairs that syslog-ng PE stores about the log message as CEF fields. Using value-pairs, you can:
-
select which value-pairs to use as CEF fields,
-
add custom value-pairs as CEF fields,
-
rename value-pairs, and so on.
For details, see the section called “Structuring macros, metadata, and other value-pairs”. Note that the syntax of format-*
template functions is different from the syntax of value-pairs()
: these template functions use a syntax similar to command lines.
Using the format-cef-extension
template function, has the following prerequisites:
-
Load the the cef
module in your configuration:
@module cef
-
Set the on-error
global option to drop-property
, otherwise if the name of a name-value pair includes an invalid character, syslog-ng PE drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)
options {
on-error("drop-property");
};
-
The log messages must be encoded in UTF-8. Use the encoding()
option or the validate-utf8
flag in the message source.
Example 14.4. Using the format-cef-extension
template function
The following example selects every available information about the log message, except for the date-related macros (R_*
and S_*
), selects the .SDATA.meta.sequenceId
macro, and defines a new value-pair called MSGHDR
that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
$(format-cef-extension --scope syslog,all_macros,selected_macros \
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
--pair MSGHDR=\"$PROGRAM[$PID]: \")
The following example selects every value-pair that has a name beginning with .cef.
, but removes the .cef.
prefix from the key names.
template("$(format-cef-extension --subkeys .cef.)\n")
The following example shows how to use this template function to store log messages in CEF format:
destination d_cef_extension {
file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
};
Syntax:
$(format-json parameters)
Description: The format-json
template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json
, you can reformat any incoming message as JSON.
You can use the value-pairs that syslog-ng PE stores about the log message as JSON fields. Using value-pairs, you can:
-
select which value-pairs to use as JSON fields,
-
add custom value-pairs as JSON fields,
-
rename value-pairs, and so on.
For details, see the section called “Structuring macros, metadata, and other value-pairs”. Note that the syntax of format-json
is different from the syntax of value-pairs()
: format-json
uses a syntax similar to command lines.
Example 14.5. Using the format-json template function
The following example selects every available information about the log message, except for the date-related macros (R_*
and S_*
), selects the .SDATA.meta.sequenceId
macro, and defines a new value-pair called MSGHDR
that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
$(format-json --scope syslog,all_macros,selected_macros \
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
--pair MSGHDR=\"$PROGRAM[$PID]: \")
The following example shows how to use this template function to store log messages in JSON format:
destination d_json {
file("/var/log/messages.json" template("$(format_json --scope selected_macros --scope nv_pairs)\n"));
};
|
NOTE:
In case of syslog-ng macros starting with a dot (for example ".SDATA.meta.sequenceID ") an empty key name is added at the top level of the JSON structure. You can work around this by adding --shift 1 as a parameter to the template function. For example in case of ".SDATA.meta.sequenceID ", an empty key name is added at the top level of the JSON structure: {"":
{"SDATA" :
{"meta" :
{"sequenceID": "123"}
}
}
} |
This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value
elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value"
. For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.
To select which value-pairs to convert, use the command-line syntax of the value-pairs()
option. For details on selecting value-pairs, see the section called “value-pairs()”.
Example 14.6. Using the format-welf()
template function
The following example selects every available information about the log message, except for the date-related macros (R_*
and S_*
), selects the .SDATA.meta.sequenceId
macro, and defines a new value-pair called MSGHDR
that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
$(format-welf --scope syslog,all_macros,selected_macros \
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
--pair MSGHDR=\"$PROGRAM[$PID]: \")
The following example shows how to use this template function to store log messages in WELF format:
destination d_welf {
file("/var/log/messages.welf" template("$(format-welf --scope selected_macros --scope nv_pairs)\n"));
};
Syntax:
$(grep condition value-to-select)
Description: The grep
template function is useful when using a pattern database to correlate related log messages. The grep
template function can be used to filter the messages of the same context when the index of the particular message is not known.
Example 14.7. Using the grep template function
The following example selects the message of the context that has a username
name-value pair with the root
value, and returns the value of the auth_method
name-value pair.
$(grep ("${username}" == "root") ${auth_method})
It is possible to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep
, these will be returned also separated by commas. This can be used for example to collect the e-mail recipients from postfix messages.
Syntax:
$(<method> [opts] $arg1 $arg2 $arg3...)
Options:
--length N, -l N
Truncate the hash to the first N characters.
Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.
<method>
can be one of md5, md4, sha1, sha256, sha512 and "hash", which is equivalent to sha256. Macros are expected as arguments, and they are concatenated without the use of additional characters.
This template function can be used for anonymizing sensitive parts of the log message (for example username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.
Also, using this template, quasi-unique IDs can be generated for data, using the --length
option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.
Example 14.8. Using the $(hash) template function
The following example calculates the SHA1 hash of the hostname of the message:
$(sha1 $HOST)
The following example calculates the SHA256 hash of the hostname, using the salted
string to salt the result:
$(sha1 $HOST salted)
To use shorter hashes, set the --length
:
$(sha1 --length 6 $HOST)
To replace the hostname with its hash, use a rewrite rule:
rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};
Example 14.9. Anonymizing IP addresses
The following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:
rewrite pseudonymize_ip_addresses_in_message {
subst (
"((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))",
"$(sha1 $0)",
value("MSG"),
flags(global)
);
};
Syntax:
$(if (<condition>) <true template> <false template>)
Description: Returns the value of the <true template>
parameter if the <condition>
is true. If the <condition>
is false, the value of <false template>
is returned.
Example 14.10. Using pattern databases and the if template function
The following example returns violation
if the username
name-value pair of a message processed with pattern database is root
, and system
otherwise.
$(if ('${username}' == 'root') 'violation' 'system')
This can be used to set the class of a message in pattern database rules based on the condition.
<value name="username">$(if ("${username}" == "root") "violation" "system")</value>
Since template functions can be embedded into each other, it is possible to use another template function as the template of the first one. For example, the following expression returns root
if the username is root
, admin
if the username is joe
, and normal user
otherwise.
<value name="username"> $(if ("${username}" == "root")
"root"
$(if ("${username}" == "joe") "admin" "normal user")
"normal user")</value>
Syntax:
$(indent-multi-line parameter)
Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.
Example 14.11. Using the indent-multi-line template function
The following example writes multi-line messages into a text file.
destination d_file {
file ("/var/log/messages"
template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );
};
Syntax:
$(ipv4-to-int parameter)
Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.
|
NOTE:
This template function is available only if the convertfuncs module has been loaded. By default, syslog-ng PE loads every available module.
By default, syslog-ng PE loads every available module. For details, see the section called “Loading modules” |
Syntax:
$(<operator> <first_operand> <second_operand>)
Description:
This template function performs simple numerical operations (like addition or multiplication) on integer numbers or macros containing integer numbers (for example, ${LEVEL_NUM}
or ${YEAR}
), and returns the result of the operation. The values used in the operation are not modified, that is, macros retain their original values. If one of the operands is not an integer, syslog-ng PE will not execute the template function, and return the NaN
(Not-a-Number) value.
Available only in syslog-ng PE 4 F1 and later.
|
Caution:
The output of this template function is always an integer. If the return value would be a floating-point number, the floating part is simply omitted, for example, 5.2 becomes 5, 5.8 becomes 5, -5.8 becomes -5. |
It is also possible to nest numerical operations.
Example 14.12. Using numerical template functions
The following template function returns the facility of the log message multiplied by 8:
$(* ${FACILITY_NUM} 8)
Template functions can be nested into each other: the following template function returns the facility of the log message multiplied by 8, then adds this value to the severity of the message (the result is actually the priority of the message):
$(+ ${LEVEL_NUM} $(* ${FACILITY_NUM} 8))
Syntax:
$(strip "<macro>")
Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:
$(strip "${MESSAGE}" "${PROGRAM}")
Available in syslog-ng PE version 6.0.6 and later.